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Preface 


This  report  addresses  Phase  n  of  a  two-phase  research  effort  to  analyze 
Squadron  Level  Training  (SLT)  in  the  Air  Force. 

Phase  I  focused  on  pilots  in  flying  units  and  enlisted  personnel  in 
maintenance  units.  This  initial  effort  involved  1)  convening  a  meeting  of 
training  experts  for  identifying  unit  level  training  requirements,  current  levels  of 
training  deficiency,  identifying  centralized  training  transferable  to  the  squadron 
level,  identifying  existing  and  needed  technology  applications,  and  developing  a 
preliminary  training  requirements  and  technology  match  ("SLT  taxonomy");  2) 
development  of  a  structured  interview  questionnaire  and  data  collection  visits;  3) 
analysis  of  data;  and  4)  refinement  of  the  SLT  taxonomy  based  on  data  collected. 

The  scope  of  Phase  II  is  to  I)  refine  the  research  methodology  developed  in 
Phase  I,  including  data  collection  and  data  analysis  methods  for  assessing  unit 
level  training  in  both  peacetime  and  wartime  environments;  2)  develop  a 
technical  approach  for  evaluating  the  generalizability  of  training  issues  across 
different  types  of  Air  Force  units;  and  3)  develop  criteria  and  a  technical  approach 
for  assessing  the  importance  of  training  needs  and  technology  applications. 

As  documented  in  this  report,  the  results  of  Phase  II  include  a  comprehensive 
training  requirements  and  technology  match  and  a  Science  and  Technology 
Investment  Strategy  for  supporting  training  technologies  at  the  squadron  level. 


SUMMARY 


The  purpose  of  this  report  is  to  present  a  science  and  technology  investment 
strategy  that  will  address  the  development  of  training  technology  applications 
relative  to  current  and  future  training  needs  and  instructional  settings  of  various 
squadron  level  training  (SLT)  environments  and  methodologies  to  accomplish 
training. 

This  report  summarizes  the  findings  of  surveys,  interviews,  and  archival 
research  conducted  to  identify  current  and  future  training  needs  of  the  USAF  as 
they  relate  to  management,  development,  delivery,  and  evaluation  of  squadron 
level  training.  These  findings  are  used  as  the  basis  for  the  identification  of 
technologies  that  can  be  applied  to  address  training  needs. 

On  the  basis  of  the  research  conducted,  a  comprehensive  investment  strategy 
is  presented  in  this  report  which  provides  recommendations  and  guidance 
regarding  the  structure  and  content  of  future  research  activities  related  to 
squadron  level  training  conducted  by  the  Technical  Research  Division,  Human 
Resources  Directorate  (HRD),  Armstrong  Laboratory,  Brooks  AFB,  TX. 

It  is  possible  to  conceive  a  world  with  infinite  time  and  non-depletable 
resources.  In  such  a  world,  researchers  would  have  an  infinite  amount  of  time  to 
research  and  develop  all  that  they  desire.  There  would  be  no  need  to  prioritize; 
nothing  would  have  to  be  foregone.  In  such  a  world,  nothing  would  be  scarce. 

In  the  real  world,  all  important  resources  are  finite.  Researchers  must  make 
decisions  among  the  various  possible  research  areas.  By  selecting  one  path  over 
another,  researchers  must  consider  the  ’’opportunity  costs,”  that  is  the  benefit,  or 
value  of  the  benefit,  that  is  foregone  by  choosing  one  alternative  over  another.  In 
a  world  of  scarcity,  deciding  which  problems  to  address  and  the  methodology  to  be 
employed  are  probably  the  most  important  decisions  made  by  a  research 
organization. 


A  Changing  Environment 

The  Air  Force  must  structure  its  training  programs  within  the  context  of  an 
ever  changing  environment.  A  number  of  trends  within  this  environment  have 
important  implications  for  the  AF  in  terms  of  training.  These  trends  include:  1) 
reduction  in  force  size,  2)  increased  skill  demands  placed  on  individuals,  3) 
continuing  importance  of  on-the-job  training  (OJT),  4)  increased  complexity  of 
weapon  systems,  5)  rapidly  changing  technology,  6)  a  diminished  national 
education  baseline,  and  7)  increased  use  of  automated  systems. 

Regardless  of  the  changes  made  in  the  Air  Force  training  structure, 
squadrons  will  continue  to  be  responsible  for  a  large  share  of  the  training  burden. 
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Trainers  at  the  squadron  level  will  continue  to  train  large  numbers  of  trainees  on 
increasingly  complex  taskj.  Consequently,  training  technologies  will  be  needed  at 
the  squadron  level  to  increase  the  efficiency  and  effectiveness  of  training  while  not 
detracting  from  the  operational  missions  of  the  units. 


History  of  the  Research 

AL/HR,  recognizing  the  importance  of  training  at  the  squadron  level  and 
assuring  that  research  activities  focus  on  specific  AF  training  needs,  implemented 
the  Squadron  Level  Training  (SLT)  project  Phase  I  focused  on  training  in  aircraft 
maintenance  units.  Phase  II,  building  on  the  findings  of  Phase  I,  expanded  the 
scope  of  the  effort  to  cover  a  wider  range  of  squadron  level  training  needs.  In 
Phase  13,  surveys  were  conducted  at  AF  bases  worldwide.  In  addition,  interviews 
focusing  on  the  SLT  training  needs  were  given  to  a  small  sample  at  six  bases  to 
verify  survey  findings  and  provide  a  depth  of  detail  that  could  not  be  as  effectively 
gathered  through  the  use  of  structured,  self-administered,  surveys. 


Overview  of  the  S&T  Investment  Strategy  Document 

The  S&T  Investment  Strategy  document  is  intended  to  provide  decision 
makers  with  recommendations  and  guidance  regarding  research  activities  related 
to  squadron  level  training.  The  S&T  Investment  Strategy  will: 

•  Identify  current  training  needs. 

•  Identify  potential /future  training  needs. 

•  Identify  existing  technologies  that  may  address  current  training  needs 
with  little  or  no  modification. 

•  Identify  potential  lines  of  research  for  the  development  of  technologies 
which  do  not  currently  exist  or  require  extensive  modification. 

•  Develop  a  proposed  research  agenda  to  address  SLT  needs. 
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RESEARCH  METHODOLOGY 

Overview 


The  research  methodology  required  a  systematic  procedure  for  gathering  data 
on  SLT  needed  to  identify  significant  areas  of  research  which  would  benefit  the  Air 
Force.  This  section  describes  in  moderate  detail  how  each  of  the  basic  activities 
were  developed  and  carried  out.  Those  readers  wishing  a  highly  detailed,  in-depth 
explanation  of  the  various  research  activities  are  invited  to  review  the 
Appendices  of  this  report.  The  research  activities  will  be  discussed  in  the 
following  order 

•  Instrument  Development 

•  The  Sampling  Plan 

•  Data  Collection 

•  D-ta  Analysis 

•  Identifying  Technologies 

•  Matching  Technologies  to  Current  and  Future  Air  Force  Needs 


Instrument  Development 

The  primary  sources  of  information  used  to  determine  squadron  level 
training  needs  were  the  three  surveys  given  to  Training  Managers,  Trainers,  and 
Trainees,  supplemented  by  interviews  and  archival  research. 

In  the  development  of  the  surveys,  the  following  steps  were  carried  out. 

•  Identify  the  population. 

•  Develop  survey  and  interview  items  that  address  the  specific  types  of 
information  identified. 

•  Pilot  test  and  refine  the  instrument. 


Identifying  the  Population 

In  order  to  identify  SLT  needs,  it  was  first  necessary  to  identify  those  who  are 
directly  affected  and  involved  in  training  at  the  squadron  level.  Preliminary 
examinations  of  the  activities  involved  in  squadron  level  training,  as  well  as  the 
results  of  the  Phase  I  assessment  of  Pacific  Air  Forces  (PACAF)  training  for  aircraft 
maintenance,  revealed  that  there  are  three  groups  of  enlisted  personnel  with 
different  levels  of  involvement  in  squadron  level  training.  These  are:  1)  Trainees  - 
the  recipients  of  training;  2)  Trainers  -  those  who  deliver  (and  sometimes 
develop)  training  at  the  local  level,  including  on-the-job  (OJT)  training 
supervisors;  and  3)  Training  Managers  -  those  who  manage  the  training  activities 
on  each  base. 
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The  following  operational  definitions  for  the  three  groups  were  used: 

•  Trainees.  Those  who  have  undergone  unit  level  training  within  the 
past  year  and  have  enough  experience  in  the  Air  Force  to  judge  the 
relevance  of  that  training  to  their  jobs. 

•  Trainers.  Trainers  include  work  center  supervisors  providing 
oversight  to  OJT,  those  providing  ancillary  training,  as  well  as  those 
providing  formal  classes  on  base. 

•  Training  Managers.  Training  Managers  are  those  who  perform  the 
administrative  and  managerial  tasks  such  as  scheduling, 
documentation  of  training  completed,  scoring  exams,  record-keeping, 
and  so  forth  to  support  squadron  level  training  on  each  base.  Training 
Managers  may  or  may  not  hold  the  75XXX  AFSC. 

A  pre-identified  sub-category  of  the  populations  of  interest  were  those  who 
had  participated  in  Desert  Shield/Desert  Storm.  This  was  felt  necessary  in  order  to 
gain  insights  on  training  in  a  wartime  environment. 


Survey  and  Interview  Items 

The  purpose  of  the  surveys  and  interviews  was  to  collect  information  for  the 
development  of  a  Science  and  Technology  Investment  Strategy  for  AL/HR  from 
which  a  report  will  be  prepared  to  address  the  development  of  training  technology 
applications  relative  to  current  and  future  training  needs  uid  instructional 
settings  of  various  SLT  environments  and  methodologies  to  accomplish  training. 

To  address  this  goal,  separate  surveys  and  interviews  were  drafted  for 
Training  Managers,  Trainers,  and  Trainees.  A  single  survey  and  interview  for  all 
respondents  would  have  been  extremely  long  and  burdensome  to  the  intended 
respondents.  Further,  responses  to  a  very  long  instrument  were  felt  likely  to  be 
unreliable,  as  respondents  could  lose  interest  or  become  annoyed  by  items  having 
low  relevance  to  their  own  situation. 

All  survey  items  were  developed  using  fixed  sets  of  responses.  For  some 
items  the  respondents  could  select  more  than  one  response;  for  others  the 
respondents  could  select  only  one  response. 
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The  items  were  grouped  in  the  following  manner 

Training  Managers  Survey: 

•  Demographics 

•  Your  function  in  managing  training 

•  Scheduling 

•  Training  development 

•  Documentation 

•  Reporting 

•  Evaluation 

•  Opinions  about  training 

•  Future  training 

•  Your  training  activities  during  Desert  Shield/Desert  Storm 
Unit  Level  Trameis.S.my.ey: 

•  Demographics 

•  OJT  trainer  /supervisor 

•  Training  development 

•  Training  delivery 

•  Opinions  about  training 

•  Future  training 

•  Your  training  activities  during  Desert  Shield/Desert  Storm 

Recipients  of  Training  Survey: 

•  Demographics 

•  Methods  of  training 

•  Upgrade  training 

•  Ancillary  training 

•  Your  training  activities  during  Desert  Shield /Desert  Storm 

When  constructing  the  surveys,  items  were  created  to  cover  each  step  of  the 
Instructional  System  Development  (ISD)  process,  as  well  as  to  explore  potential 
problem  areas  concerning  each  task  in  delivering  and  managing  unit-level 
training.  Appendix  A  lists  survey  items  against  specific  ISD  process  characteristics. 

Since  the  responsibilities  and  experiences  of  the  three  survey  groups  were  not 
the  same,  the  surveys  were  not  identical.  When  possible,  the  same  question  was 
asked  to  two  or  more  of  the  various  groups  allowing  comparison  across  groups. 
The  interview  instruments  were  developed  following  the  same  general  process  as 
that  used  for  the  surveys.  The  interview  format  used  allowed  free  response  but, 
because  we  noted  during  pilot  testing  that  most  responses  fell  into  a  limited  set  of 
possible  responses,  a  checklist  of  these  responses  was  provided.  This  was  done  in 
order  to  speed  the  interview  process  by  reducing  writing  time.  Respondents  were 
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not  cued  or  given  possible  responses,  but  were  allowed  to  respond  freely.  (Copies 
of  the  surveys  and  interviews  can  be  found  in  Appendices  B  and  C  respectively.) 


Instrument  Pilot  Test  and  Refinement 

An  initial  pilot  test  of  the  survey  items  was  conducted  at  Andrews  AFB,  MD. 
Input  from  the  participating  active  duty  personnel  helped  in  the  wording  of  items 
and  identification  of  important  general  environmental  and  administrative 
characteristics  of  training  at  the  unit  level. 

Both  the  surveys  and  interviews  were  again  tested  at  Randolph  AFB,  TX. 
Based  on  the  pilot  test  findings,  survey  questions  were  modified  and  the  interview 
format  was  significantly  modified.  During  the  pilot  test,  we  discovered  that  most 
of  the  responses  to  the  questions  could  be  classified  into  a  limited  set  of  categories. 
Taking  advantage  of  this  factor,  the  interviewer  guide  was  modified,  allowing  the 
interviewers  to  quickly  code  a  response.  If  the  response  fell  outside  the  pre-coded 
responses  space  was  available  for  the  respondents'  answers  to  be  written. 


Sampling  Plan 

The  following  criteria  guided  the  development  of  the  sampling  and  data 
collection  plans  for  the  surveys  and  interviews  conducted: 

•  The  sampling  and  data  collection  plan  must  be  administratively 
feasible,  i.e.,  within  the  constraints  of  normal  AF  operations. 

•  The  resulting  data  should  be  representative  of  the  population  of 
interest  to  AF  policy  makers. 

•  The  results  should  be  generalizable  to  the  overall  AF  enlisted 
population. 

•  The  data  should  facilitate  comparisons  among  types  of  AF  personnel 
classified  in  multiple  ways,  such  as  by  AFSC,  by  years  of  service,  by  type 
of  location,  and  so  forth,  in  order  to  examine  whether  findings  are  the 
same  or  different  for  various  types  of  personnel. 

Trainee  Sample:  A  total  of  1589  airmen  completed  the  survey  out  of  2663 
contacted.  These  were  clustered  in  the  E3  to  E6  pay  grade.  Only  two  individuals 
were  out  of  this  range,  one  airman  basic  and  one  airman.  The  expected  return  rate 
was  1000,  assuming  a  35%  return  rate.  The  actual  return  rate  was  57%,  which  is 
unusually  high  for  survey  research.  For  the  telephone  interviews  conducted,  40 
additional  Trainees  were  interviewed. 
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Trainpr  Sample.-  A  total  of  2645  potential  respondents  were  contacted  to 
complete  the  Trainer  Survey.  Of  these,  246  indicated  that  they  did  not  match  the 
requirement  set  forth  in  the  cover  letter  of  the  survey.  1379  respondents  completed 
the  Trainer  survey,  or  54%  of  those  contacted.  This  again  is  well  above  the  normal 
return  rate  for  a  survey.  An  additional  40  Trainers  took  part  in  the  telephone 
interviews.  The  pay  grade  of  the  Trainers  ranged  between  E4  and  E9.  A  cover 
memo  was  attached  to  the  survey  form,  indicating  the  types  of  individuals  who 
were  defined  as  Trainers.  Those  who  believed  they  did  not  fit  this  broad  definition 
checked  off  a  box  on  the  cover  memo  indicating  that  it  was  not  applicable,  and 
returned  the  survey  to  the  on-base  distribution  point.  Those  who  fit  the  definition 
of  Trainer  given  on  the  cover  memo  completed  the  survey  and  returned  it  to  the 
distribution  point.  The  resulting  data  are  felt  to  adequately  represent  the 
population  of  Trainers. 

Training  Manager  Sample:  The  Training  Manager  Survey  was  completed  by 
400  respondents,  with  an  additional  38  respondents  taking  part  in  the  telephone 
interviews. 

Since  Training  Managers  comprise  a  relatively  small  population  at  any  one 
base,  the  full  population  of  Training  Managers  on  the  sampled  installations  were 
asked  to  complete  the  survey  in  order  to  yield  a  sufficiently  large  sample.  From  an 
exploratory  site  visit  to  Andrews  AFB,  it  was  estimated  that  large  bases  may  have 
twenty  to  thirty  people  functioning  as  Training  Managers. 

Instructions  contained  in  the  cover  memo  of  the  Survey  Form  for  Unit  Level 
Trainers  instructed  those  Training  Managers  mistakenly  given  the  Survey  Form 
for  Unit- Level  Trainers  not  to  complete  the  Trainer’s  survey,  and  request  a  copy  of 
the  Survey  Form  for  Training  Managers  from  their  point  of  contact.  This  sample 
of  Training  Managers  constituted  a  probability  sample  of  the  Training  Managers, 
since  they  were  to  be  drawn  from  the  same  base  population  as  Trainees  and 
Trainers. 

For  this  project,  we  believed  it  was  desirable  to  include  in  the  survey. 

Training  Managers  who  were  performing  outside  of  their  primary  AF5C  in  order 
to  understand  how  often  this  occurs.  Including  these  individuals  also  revealed 
whether  the  current  uses  and  future  needs  for  training  technology  among  those 
without  formal  preparation  for  the  career  field  are  different  from  those  serving  in 
the  career  field. 


The  Sampling  Strategy 

The  returned  questionnaires  from  3,368  individuals  in  the  surveys  of 
Trainees  and  of  Trainers  provided  large  enough  samples  for  a  variety  of  sub-group 
analyses  of  the  types  indicated  above.  The  initial  samples  were  large  enough  to 
allow  for  expected  non-retumed  surveys,  and  for  a  number  of  "not  applicables" 
among  the  Trainers. 


In  constructing  the  sampling  strategy,  we  assumed  that  the  intended  target 
population  is  the  total  enlisted  Air  Force  population  who  fall  within  each  survey 
group.  Using  this  definition  of  the  target  population  and  probability-based 
methods  for  sampling,  the  survey  results  were  generalizable  to  the  total 
population  of  policy  interest.  Generalizable  comparisons  could  also  be  made  about 
the  perceptions  and  needs  for  training  of  those  within  larger  versus  smaller  bases, 
maintenance  versus  non-maintenance  occupations,  major  versus  less  populous 
AFSCs,  and  so  forth.  Given  the  focus  of  the  project  on  squadron-level  training, 
this  definition  of  the  target  population  was  desirable  because  those  with  the 
greatest  current  training  needs  may  in  fact  be  those  who  are  not  in  the  larger  bases 
or  most  populated  AFSCs. 

A  probability  sampling  procedure  was  used  to  draw  the  sample.  Probability- 
based  procedures  yield  generalizable  population  estimates  of  calculable  precision. 
With  the  use  of  probability  sampling  procedures,  the  sources  of  biases  or  non- 
generalizability  can  be  accurately  calculated. 

When  considering  population  definitions  and  sampling  strategies,  all  sub¬ 
groups  that  were  included  in  analysis  did  not  need  to  be  explicitly  created  at  the 
time  of  sampling,  i.e.,  in  a  stratified  sampling  plan.  Particularly  when  the  sample 
was  likely  to  be  analyzed  for  differences  among  a  number  of  sub-groups  (e.g.,  by 
AFSCs,  by  years  of  service,  by  grade  level,  by  size  of  current  installation,  by  types  of 
specific  responsibilities  for  training,  by  those  with  or  without  certain  prioi 
experiences,  etc.),  it  is  usually  preferable  not  to  stratify  the  sample  a  priori.  Instead, 
the  sample  should  be  drawn  to  be  representative  of  the  total  defined  population, 
then  various  sub-groups  are  examined  during  the  analysis  stage. 

A  two-stage  sampling  technique  was  used.  Air  Force  installations  were 
sampled  with  a  probability  proportional  to  their  size  (i.e.,  larger  installations 
having  a  higher  probability  of  being  in  the  final  sample).  Then,  individuals  who 
met  the  criteria  for  the  Trainee  and  Trainer  groups  were  sampled  within  the 
selected  installations  using  a  uniform  sampling  fraction  across  installations,  so 
that  larger  installations  had  proportionally  larger  shares  of  the  final  samples.  The 
Training  Manager  sample  was  file  full  population  of  Training  Managers  within 
the  sampled  installations.  This  sampling  strategy  required  actual  data  collection 
only  at  sampled  installations,  which  permitted  the  use  of  on-base  data  collection 
procedures.  It  was  necessary'  to  limit  the  initial  selection  of  bases  to  those  above  a 
certain  minimum  size  (only  bases  with  more  than  1,000  enlisted),  to  avoid  having 
a  large  number  of  installations  with  only  a  few  sampled  individuals  in  the  study. 

The  Air  Force  Military  Personnel  Center  Survey  Branch  was  consulted  and 
confirmed  that  the  statistical  and  operational  support  for  this  type  of  sampling  was 
obtainable.  The  sampling  of  installations,  and  of  individual  Trainees  and  Trainers 
was  completed  by  the  Armstrong  Laboratory  using  the  personnel  records  data  files. 
These  procedures  generated  lists  of  sampled  individuals  for  the  Trainee  and 
Trainer  surveys,  organized  by  installation. 
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Data  Collection 

Current  problems  and  future  needs  identified  in  this  report  are  primarily 
based  on  the  results  of  analysis  of  the  survey  and  interview  data  collected. 

The  Technical  Training  Research  Division,  Human  Resources  Directorate  of 
the  Armstrong  Laboratory  directly  managed  the  distribution  and  collection  of  the 
surveys.  Analysis  was  performed  under  contract  by  Hay  Systems,  Inc. 

On-base  data  collection  was  used  for  the  survey  and  interviews.  The  Survey 
Control  Officer  (SCO)  on  each  base  managed  the  distribution  of  surveys,  with 
assistance  from  the  Base  Training  Manager  when  needed  to  identify  Training 
Managers.  The  individually  addressed  survey  questionnaires,  answer  sheets,  and 
names  of  sampled  persons  for  each  type  of  survey  on  that  base  were  mailed  to  the 
SCO.  The  SCO  was  responsible  for  contacting  the  sampled  individuals  and 
obtaining  the  completed  survey  form.  This  on-base  data  collection  procedure  was 
particularly  appropriate  with  a  two-stage  sampling  plan  used.  Using  an  on-base 
SCO  also  provided  much  more  assurance  of  a  good  response  rate. 

The  telephone  interviews  were  a  small  sample  study  conducted  with 
respondents  identified  and  scheduled  in  advance  at  six  CONUS  bases.  No 
significant  problems  were  encountered  in  collecting  data  using  this  approach. 

Data  Analysis 

Analysis  Software 

In  order  to  perform  statistical  analysis  on  the  data  collected,  the  SPSS 
statistical  analysis  package  was  used.  This  package  allowed  rapid  exploration 
analysis  and  preparation  of  tables  and  charts  directly  from  the  data. 

Lay-out  of  Analytic  Data  Files 

The  creation  of  analytic  data  files  from  the  OpScan  form  was  done  by  data 
processing  personnel  at  the  Armstrong  Laboratory.  Data  fields  were  of  fixed 
length,  from  the  OpScan  input.  For  items  that  are  of  the  "Mark  all  that  apply" 
type,  a  separate  data  field  was  used  for  each  alternative,  entering  a  "1"  if  checked, 
and  ”0"  if  not  checked.  This  permitted  easy  analysis  and  simplified  the  creation  of 
summary  variables,  such  as  the  number  of  items  checked  on  the  question.  For 
items  with  a  response  scale,  e.g.,  "strongly  agree"  to  "strongly  disagree",  the 
response  letters  were  converted  to  a  numeric  1  to  5  scale,  to  permit  quantitative 
analysis. 

Explicit  conventions  were  set  up  for  handling  missing  data  elements  and  the 
"don't  know/not  applicable"  responses.  The  Data  Analysis  Code  Sheet  in 
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Appendix  D  provides  a  detailed  description  of  each  item  meaning  location  in  the 
data  string  and  conventions  used. 


Data  Cleaning 

After  creating  each  analytical  file,  numerical  data  was  examined  for  any  illegal 
values,  or  extreme  outliers.  Outliers  found  were  checked  against  other  data 
elements  in  the  record,  or  in  reference  to  the  personnel  data  and  corrections  were 
made  if  feasible.  Changing  illegal  values  to  a  missing  data  code  was  a  last  resort. 
Overall  the  data  files  provided  by  the  Armstrong  Laboratory  contained  remarkably 
few  errors. 


Analysis 

The  methodology  used  for  the  analysis  of  the  data  is  described  in  detail  in 
Appendix  E,  the  Squadron  Level  Training  Data  Processing  and  Data  Analysis  Plan. 

The  following  steps  in  data  analysis  were  used  for  each  form  as  its  data 
became  available  after  data  entry  and  cleaning.  With  this  type  of  exploratory 
survey  data  analysis,  the  range  of  specific  analyses  to  be  performed  is  very  large,  so 
the  expected  outcomes  could  not  be  predicted  in  advance. 

Step  1  -  Frequencies.  The  numbers  and  percentages  for  the  full  set  of 
responses  for  each  item  were  examined.  The  frequencies  and  percentages  for 
many  items  were  of  interest  in  themselves,  such  as  the  types  of  training 
experienced,  delivered  or  managed;  the  perceptions  of  problems  encountered;  the 
types  of  training  technology  used;  and  opinions  about  the  current  unit-level 
training  situation. 

Step  2  -  Create  Summary  Variables.  The  use  of  summary  variables  made  the 
data  easier  to  understand  These  variables  tend  to  be  of  two  types:  1)  refined  sets  of 
categories  to  be  used  in  er  cross-tabulations,  such  as  the  AFSC  job  category;  or  2) 
continuous  variables  measuring  something  on  a  numeric  scale,  such  as  a 
summary  index  of  problems  perceived  with  a  specific  type  of  training. 

Step  3  -  Cross-Tabulations  and  Comparisons  of  Means.  The  analyses  helped 
to  understand  whether  respondents  from  different  backgrounds  or  with  different 
experiences  in  the  Air  Force  tend  to  react  differently  to  the  survey  items.  A  cross¬ 
tabulation  is  a  table  for  examining  the  relationship  of  one  categorical  variable  to 
another  categorical  variable,  usually  using  the  X2  statistic  for  testing  the  presence  of 
an  overall  relationship.  For  example,  do  Trainees  from  different  AFSCs  tend  to 
experience  different  methods  for  job-related  training?  Do  Training  Managers 
’'with"  versus  those  "without"  the  75XXX  code  as  their  primary  AFSC  tend  to  have 
different  functions  in  managing  training?  Do  they  use  technology  differently? 
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A  comparison  of  means  was  used  to  examine  differences  among  subgroups 
on  a  quantitative  item,  such  as  items  scored  from  "strongly  agree"  to  "strongly 
disagree,"  or  a  scale  combining  several  such  items.  For  example,  do  Training 
Managers  coming  from  different  types  of  squadrons  tend  to  respond  differently  to 
the  items  about  problems  with  documentation  or  scheduling?  Do  Trainees  from 
different  AFSCs  have  different  opinions  about  the  training  technology  they  have 
experienced? 

A  large  number  of  exploratory  analyses  of  this  type  were  feasible,  especially 
using  the  summary  variables  discussed  above.  The  results  of  the  most  significant 
or  surprising  of  these  analyses  are  presented  in  this  report. 

Some  of  the  sub-groups  that  were  examined  within  these  data  concerning 
Squadron  Level  Training  are  the  following: 

•  Types  of  units 

•  AFSC  job  classes,  collapsed  as  needed  into  major  groupings 

•  Training  Managers  "with"  versus  "without”  the  75XXX  AFSC 

•  Grade  level,  E-3  to  E-9  (as  relevant  for  each  survey  form) 

•  Major  Command  (MAJCOM) 

•  Years  of  active  duty  service,  categorized  as  less  than  1,  1  to  2, 2  to  3, 3  to 
4,  or  4  or  more  for  Trainees;  categorized  as  1  to  4, 5  to  8,  9  to  12, 13  to  16, 
or  more  than  16  for  Trainers  and  Training  Managers 

•  Role  in  the  training  process  for  Trainers  and  Managers 

•  Training  Managers  who  spend  most/all  of  their  duty  time  on  training 
activities  versus  those  who  spend  little  of  their  duty  time  on  training 

•  Trainees  who  have  experienced  different  types  of  training  technologies 

Step  4  -  Correlational  Analysis.  This  technique  was  used  to  understand 
relationships  among  continuous  items,  and  to  analyze  more  complex  patterns  of 
relationships.  Since  these  surveys  were  intended  primarily  to  obtain  descriptive 
information  about  current  squadron-level  training  practices  and  perceived  needs 
for  the  future,  the  items  were  not  designed  to  test  particular  hypotheses  or  causal 
models.  Nevertheless,  some  exploratory  analyses  of  potential  relationships  were 
feasible. 
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Specific  Analytic  Questions 


The  analytic  steps  outlined  above  all  involve  using  the  data  from  these  three 
groups  of  respondents  to  understand  the  current  extent  of  use  of  technologies  for 
unit  level  training,  and  respondents'  assessments  of  the  effectiveness,  reliability, 
problems,  or  other  applicable  dimensions  associated  with  that  aspect  of  training. 
Interpreting  the  nature  of  the  respondents'  perceptions  in  light  of  the  extent  of 
their  experience  with  a  technology  provided  information  for  making  inferences 
concerning  a  match  between  current  and  future  needs  and  technologies. 

Most  sections  of  the  surveys  and  interviews  examined  a  specific  topic 
concerning  training,  such  as  OJT,  ancillary  training,  or  management  functions 
such  as  scheduling,  record-keeping  or  evaluation.  This  allowed  the  use  of  the 
following  general  logic  to  examine  topic  areas  given  with  their  corresponding 
specific  analytic  questions  (AQ): 

A.  What  training  has  been  experienced  or  which  sub-tasks  does  the 
respondent  do,  in  regard  to  this  topic? 

AQ:  What  is  the  respondent's  role/experience  with  this  Topic? 

B.  What  is  the  respondent's  scope  of  use  (e.g.,  %  time  on  that  activity,  #  years 
of  experience)  with  that  topic? 

C:  What  technologies /methods  have  been  used  for  this  topic? 

AQ:  How  extensive  is  the  current  use  of  training  technologies  in 
comparison  with  the  use  of  other  methods  (manual,  one-on-one 
instruction,  etc.)? 

AQ:  Does  the  extent  of  use  of  technologies  versus  other  methods  differ 
for  the  various  topics?  Differ  among  types  of  respondents? 

AQ:  Does  the  extent  of  use  of  training  technologies  in  comparison 
with  other  methods  differ  by  the  respondents’  roles  or  scope  of  use 
(i.e.,  items  A  &  B)? 

D.  What  are  the  respondents'  assessments  of  the  effectiveness,  reliability, 
types  of  problems,  etc.  concerning  that  topic  and/or  their  use  of  technologies 
for  it? 


AQ:  What  are  the  major  problems  (needs)  perceived  at  the  squadron 
level  with  local  level  training? 

AQ:  Are  the  assessments  or  problems  (needs)  related  to  the  extent  of 
use  of  a  specific  method? 
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AQ:  Are  the  reported  assessments  or  problems  (needs)  related  to  the 
respondent's  role  or  scope  of  use  on  that  topic? 

Since  each  of  the  topic  areas  may  also  differ  among  one  or  more  of  the 
various  sub-groups,  a  large  number  of  exploratory  data  tables  were  examined  to 
determine  whether  the  finding  for  a  topic  differed  by  sub-group  or  could  be  safely 
generalized  across  the  respondents  to  each  survey.  The  presentation  of  reported 
results  include  only  those  sub-group  analyses  that  show  meaningful  differences 
among  sub-groups. 


Identification  of  Technologies 

The  use  of  the  data  analyzed  for  matching  current  and  future  needs  with 
currently  available  mature  technologies,  and  those  technologies  currently  under 
development  which  are  expected  to  play  a  more  important  roll  in  the  future,  was 
based  on  logical  interpretations  of  the  findings  in  relation  to  the  potential  uses  and 
characteristics  of  technologies  -  both  those  currently  available  and  feasible  future 
developments.  For  this  logical  analysis,  additional  information  about  the  nature 
of  the  technologies  was  needed  to  compare  with  the  findings  from  the  surveys. 

Those  problems  and  needs  that  appeared  to  be  addressable  through  some 
form  of  technology  were  prioritized  and  the  general  characteristics  of  technologies 
that  can  address  each  problem  and  need  were  then  identified.  Research  was  then 
conducted  to  determine  which  problems  and  needs  are  currently  being  addressed 
in  the  Air  Force  through  an  existing  program,  or  for  which  research  is  currently 
conducted,  or  is  in  the  process  of  implementation.  Those  identified  as  belonging 
to  this  category  were  then  documented.  Current  and  future  needs  not  being 
addressed  by  current  AF  research  were  identified  and  documented. 

The  identification  of  technologies  that  could  potentially  benefit  squadron 
level  training  required  a  review  of  a  broad  range  of  technologies  and  their  current 
and  proposed  applications.  Some  of  the  available  sources  of  information  of 
potentially  useful  technologies  were: 

•  Current  research  and  development  activities  within  the  Air  Force 

•  Other  services'  research  and  development  activities 

•  Professional  journals  and  publications  dealing  with  technology  and 
training 

•  Subject  matter  experts 

•  University  sponsored  research 
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The  initial  identification  of  potentially  useful  technologies  is  based  on  two 
questions: 


•  What  technologies  can  be  applied  to  solve  current  or  potential 
problems  faced  by  the  Air  Force  in  squadron  level  training? 

•  What  technologies  may  benefit  and  improve  squadron  level  training 
even  though  no  problem  has  been  identified? 

In  addressing  the  First  question,  the  researchers  first  identified  problems  and 
needs.  The  survey  and  interview  phases  of  this  project  provide  the  basis  for  the 
identification  of  problems  and  needs.  Project  researchers  then  identified  generic 
characteristics  required  of  a  technology  to  address  an  identified  problem  or  need. 

Addressing  the  second  question  requires  a  broader  perspective.  Because  of  the 
rapid  advances  being  made  in  technology,  the  possibility  exists  that  a  technology  or 
a  specific  application  of  a  technology  could  radically  change  approaches  to  training 
and  management  at  the  squadron  level. 

Numerous  examples  exist  in  various  fields  where  a  technology  and  its 
application  has  had  revolutionary  effects.  The  advent  of  integrated  circuits  (ICs) 
increased  the  reliability  and  decreased  the  cost  of  practically  every  electrical 
component  used  today.  Without  ICs,  the  low  cost  personal  computer  would  have 
never  existed.  Liquid  crystal  displays,  in  combination  with  ICs,  have  lowered  the 
cost  of  watches  to  such  an  extent  the  price  of  most  digital  watches  is  primarily 
based  on  aesthetic  value  rather  than  accuracy. 

In  investigating  technologies  that  could  improve  training  even  though  no 
problem  exists,  the  researchers  first  investigated  the  technology  and  decided  if  it 
could  be  applied  to  training.  By  applying  the  criteria  for  evaluation  described  in 
this  report  and  contained  in  the  Science  and  Technology  Checklist,  some 
technologies  were  identified  that  could  prove  to  be  of  considerable  value  to  Air 
Force  training. 


Matching  Technologies  to  Current  and  Future  SLT  Needs 


Screening  Technologies  Through  the  ISD  Model 

After  the  initial  selection  of  potentially  useful  technologies,  screening  of  the 
technologies  was  required.  The  evaluation  criteria  and  strategies  used  for  initial 
screening  of  potentially  useful  technologies  were  to  be  those  proposed  in  Chin, 
Wimpee,  Laue,  Pedersen,  and  Green  (1992).  The  framework  used  was  the  end 
product  of  repeated  reviews,  modification,  and  refinement  by  AL/HRT  ISD 
experts. 
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The  use  of  this  ISD-based  framework  has  several  features  which  make  it 
appropriate  for  the  evaluation  of  technologies  as  they  apply  to  training 
development,  delivery,  and  management.  The  ISD  model  covers  a  wide  range  of 
training  activities,  including  planning,  programming,  development,  delivery, 
management,  and  evaluation.  The  ISD  model  can  be  defined  so  as  to  differentiate 
between  the  unique  functions  of  various  training  technologies.  The  overall 
structure  and  concept  of  ISD  is  widely  accepted  within  the  AF  and  other  training 
communities.  The  use  of  this  model  allowed  closer  comparison  of  Phase  II 
findings  with  those  from  the  Phase  I  research. 

The  three  survey  instruments,  when  taken  together,  cover  the  major  aspects 
of  the  ISD  process.  No  single  survey  addresses  all  ISD  functions,  since  the 
personnel  surveyed  are  involved  or  could  likely  be  involved  in  only  some  of  the 
aspects  of  the  ISD  process.  Trainees,  for  example,  would  not  be  involved  in 
defining  job  requirements,  developing  objectives,  or  validating  instructional 
materials.  Consequently,  asking  questions  related  to  these  topics  from  an  audience 
of  Trainees  would  not  yield  useful  information. 

Appendix  A  provides  a  listing  of  the  major  ISD  process  categories  and  the 
specific  survey  items  that  address  the  process.  The  appendix  does  not  list 
demographic  items  w'  {cl.  were  used  to  categorize  process-specific  requirements. 

Since  Phase  II  of  this  study  incorporated  studies  of  technologies  and  research 
outside  of  the  AF,  the  findings  of  Phase  II  could  be  expected  to  vary  somewhat 
from  that  of  Phase  I.  Nevertheless,  Phase  II  research  confirms  many  of  the 
findings  of  Phase  I.  (See  Appendix  F,  Relating  Training  Systems  Needs  to  Training 
Technologies,  for  a  discussion  of  the  evaluation  criteria  and  strategies  as  used  in 
Phase  I.) 

The  ISD-based  framework  served  as  the  initial  "screen"  to  identify  those 
technologies  which  were  further  analyzed  as  to  their  appropriateness  to  training 
needs  through  the  use  of  the  Science  and  Technology  Investment  Strategy 
Checklist. 


Prioritize  Findings 

Current  problems  and  future  needs  identified  through  quantitative  and 
qualitative  analyses  of  the  data  gathered  were  then  prioritized.  The  initial 
prioritization  was  based  on  generalizability  of  the  problem  and  the  perceived 
importance  to  overall  squadron  level  training. 


Review  of  the  Listed  Problems  and  Need 

The  list  of  current  problems  and  future  needs  was  reviewed  by  the  research 
staff.  As  part  of  this  review,  current  problems  and  future  needs  were  grouped  into 
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clusters  of  related  problems  that  may  be  addressed  through  a  single  technology  or 
related  technologies.  Those  problems  and  needs  that  did  not  seem  to  be 
addressable  through  technology  were  identified  and  eliminated  from 
consideration.  The  reasons  for  their  non-selection  are  discussed  in  this  report. 


Preparing  an  S&T  Checklist 

For  each  of  the  remaining  problems  and  needs,  a  Science  and  Technology 
Investment  Strategy  Checklist  was  prepared.  The  S&T  Checklist,  used  as  a  tool  in 
this  study,  goes  well  beyond  the  identification  of  technologies.  It  is  a  listing  of 
specific  questions  and  topics  that  need  to  be  addressed  in  the  selection,  program 
design,  research,  and  implementation  stages.  The  use  of  the  S&T  Checklist  assures 
that  important  topics  to  the  development  and  implementation  of  a  technology  are 
addressed.  As  part  of  the  Investment  Strategy,  the  checklist  was  used  to  identify 
factors  which  could  affect  the  success  or  failure  of  a  line  of  research.  We  define 
success  as  the  implementation  and  use  of  the  products  or  findings  of  research.  The 
S&T  Checklist  and  its  companion  documents  can  be  found  at  Appendices  H,  I,  and 

J. 
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SIGNIFICANT  FINDINGS  OF  SURVEYS  AND  INTERVIEWS 


A  large  quantity  of  useful  information/ data  was  collected  through  the 
administration  of  the  surveys  and  interviews  previously  discussed,  and  numerous 
analyses  were  performed.  This  section  will  describe  the  demographics  of  the 
various  sample  populations,  and  the  results  of  the  analyses  which  appear  to  be  of 
most  significance  to  the  project. 


Demographic  Characteristics  of  Samples 

Although  the  survey  was  administered  Air  Force-wide,  approximately  one- 
half  of  the  Training  Manager,  Trainer,  and  Trainee  respondents  were  from  MAC, 
SAC,  and  TAC.  (Note  that  the  survey  was  administered  prior  to  the  activation  of 
Air  Force  Combat  Command,  Air  Mobility  Command,  and  Air  Force  Materiel 
Command.)  In  addition,  a  large  portion  of  the  surveys  were  administered  in  the 
overseas  commands  of  PACAF  and  USAFE.  Unfortunately,  MAJCOM 
identification  was  structured  for  administration  of  the  survey  in  CONUS  only. 

The  survey  was  subsequently  administered  overseas  as  well,  forcing  all  overseas 
respondents  to  respond  "Other"  to  their  parent  MAJCOM.  The  survey  sample  was 
also  heavily  weighted  by  type  unit  of  assignment.  Over  one  half  of  the 
respondents  in  each  category  were  assigned  to  one  of  six  type  units:  Aircraft 
Maintenance,  Civil  Engineering,  Communications,  Security  Police,  Supply,  and 
Training.  The  "Training"  unit  of  assignment  may  be  somewhat  misleading,  in 
that  some  who  selected  this  option  may  actually  be  in  Training  sections  of  other 
type  units  such  as  Aircraft  Maintenance  or  Civil  Engineering,  etc. 

The  interviews  were  conducted  with  personnel  from  MAC,  SAC,  and  TAC 
only.  The  interviews  were  conducted  telephonically  with  Training  Managers, 
Trainers,  and  Trainees  from  MAC'S  Pope  AFB  and  Scott  AFB,  SACs  Offutt  AFB 
and  McConnell  AFB,  and  TAC's  Langley  AFB  and  Nellis  AFB. 

The  distribution  of  both  survey  respondents  and  interviewees  by  MAJCOM 
and  unit  of  assignment  may  be  found  in  Appendix  G. 
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Grades  of  Survey  Respondents 

There  were  400  Training  Managers,  1379  Trainers,  and  1589  Trainees  who 
responded  to  the  survey.  Table  1  reflects  the  grade  distribution  of  the  respondents. 


Table  1.  Grades  of  Survey  Respondents 


Training 

Managers 

Trainers 

Trainees 

# 

% 

# 

% 

# 

% 

CMS 

2 

05 

2 

0.1 

0 

0.0 

SMS 

7 

15 

21 

15 

0 

0.0 

MSG 

74 

185 

575 

41.8 

0 

0.0 

TSG 

107 

26.8 

756 

54.9 

46 

Z9 

SSG 

149 

37.3 

17 

12 

627 

395 

SGT/SRA 

55 

13.8 

5 

0.4 

645 

40.6 

A1C 

0 

0.0 

0 

0.0 

269 

16.9 

AMN 

0 

0.0 

0 

0.0 

1 

0.1 

AB 

0 

0.0 

0 

0.0 

1 

C.1 

CIV 

6 

15 

0 

0.0 

UNK 

3 

As  can  be 
five  percent  of 


seen,  this  was  a  relatively  experienced  group  of  respondents.  Eighty- 
them  had  four  or  more  years  of  service. 
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Grades  of  Interviewees 


There  were  38  Training  Managers,  40  Trainers,  and  40  Trainees  interviewed.  Table 
2  reflects  the  grade  distribution  of  the  interviewees. 


Table  2.  Grades  of  Interviewees 


Training 

Managers 

wmm 

Trainees 

# 

% 

# 

& 

# 

% 

MSG 

6 

15.8 

3 

7.5 

0 

0.0 

TSG 

7 

18.4 

11 

27.5 

0 

0.0 

SSG 

17 

44.7 

14 

35.0 

0 

0.0 

SGT/SRA 

6 

15.8 

10 

25.0 

3 

73 

A1C 

0 

0.0 

2 

5.0 

21 

52.5 

AMN 

0 

0.0 

0 

0.0 

15 

373 

AB 

0 

0.0 

0 

0.0 

1 

2.5 

CIV 

2 

53 

0 

0.0 

0 

0.0 

Overall,  the  interviewees  were  generally  more  junior  than  the  survey 
respondents. 


Other  Interviewee  Demographics 

Training  Managers  and  Trainers  who  were  interviewed  were  asked  if  that 
was  their  primary  or  additional  duty.  Table  3  reflects  their  responses. 


Table  3.  Percent  Training  Primary/ Additional  Duty 


Training  Managers 

Trainers 

Primary  Duty 

87% 

20% 

Additional  Duty 

13% 

80% 

In  a  related  question.  Trainers  were  asked  if  they  were  instructor-qualified. 
Only  30  percent  indicated  they  were. 


Significant  Survey  Findings 

Copies  of  the  survey  instruments  used,  annotated  with  frequency  response 
percentages  to  all  questions  may  be  found  at  Appendix  B.  Following  are  the 
significant  findings  regarding  degree  of  automation,  time  spent  on  training  duties, 
and  training  problems  identified. 


Training  Manager  Significant  Findings  Regarding  Degree  of  Automation 

The  following  question  was  among  those  asked  of  Training  Managers  on  the 
survey: 

Q.  10:  How  would  you  describe  the  systems  you  currently  use  to  do  your 
scheduling? 

a.  Fully  automated 

b.  Automated  scheduling  from  manual  input 

c  Manual  scheduling  from  automated  input  (source  documents) 

d.  Full}  manual 

Figure  1  reflects  the  percentages  of  all  Training  Managers  who  responded  as 
indicated. 


ALTO  AUTO/MAN  MAN/AUTO  MAN 
IN  IN 

Figure  1.  Scheduling  Degree  of  Automation 


Aircraft  Maintenance  was  among  the  most  automated  for  scheduling 
training,  and  Civil  Engineering  and  Supply  were  among  the  least  automated. 
These  types  of  units  were  examined  by  selected  MAJCOMs.  The  results  are 
displayed  in  the  next  three  figures. 
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Figure  2  reflects  the  responses  to  question  10  by  Aircraft  Maintenance 
Training  Managers. 


AUTO  AUTO/  MAN/  MAN 
MAN  IN  AUTO  IN 


Figure  2.  Aircraft  Maintenance  Scheduling  Degree  of  Automation 

SAC  and  TAC  appear  to  be  the  most  automated  in  scheduling  Aircraft 
Maintenance,  and  MAC  the  least. 

Figure  3  reflects  the  responses  to  question  10  by  Civil  Engineering  Training 
Managers. 


AUTO  AUTO/  MAN/  MAN 
MAN  IN  AUTO  IN 

Figure  3.  Civil  Engineering  Scheduling  Degree  of  Automation 

With  the  exception  of  a  small  amount  of  automation  in  MAC,  the  Civil 
Engineering  scheduling  systems  of  these  MAJCOMs  are  essentially  manual. 
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Figure  4  reflects  the  responses  to  question  10  by  Supply  Training  Managers. 


■  MAC  ■  SfiC  ■  T PC  ■  OTHER 
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Figure  4.  Supply  Scheduling  Degree  of  Automation 

In  Supply,  some  scheduling  automation  appears  to  exist  in  SAC,  but  none 
exists  in  MAC. 

Overall,  it  appears  that  SAC  has  done  the  most  in  terms  of  automated 
scheduling,  and  MAC  the  least 

We  then  examined  each  MAJCOM  individually  to  observe  the  degree  of 
automated  scheduling  by  selected  units  of  assignment. 

Figure  5  reflects  the  degree  of  scheduling  automation  in  MAC. 


■  ACFTMA1NT  ■  CIV  ENG  ■  SUP 
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Figure  5.  MAC  Degree  of  Scheduling  Automation 

What  little  scheduling  automation  that  exists  in  MAC  is  found  in  Aircraft 
Maintenance,  with  a  small  amount  indicated  in  Civil  Engineering.  None  exists  in 
Supply. 


AUTO  AUTO/MAN  MAN/  MAN 
IN  AUTO  IN 
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Figure  6  reflects  the  degree  of  scheduling  automation  in  SAC. 
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Figure  6.  SAC  Degree  of  Scheduling  Automation 

SAC  appears  to  be  highly  automated  for  scheduling,  particularly  in  Aircraft 
Maintenance.  In  addition.  Supply  appears  to  be  surprisingly  high. 

Figure  7  reflects  the  degree  of  scheduling  automation  in  TAC. 


■  ACFTMAINT  ■  CJVBvJG  ■  SUP 
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IN  IN 

Figure  7.  TAC  Degree  of  Scheduling  Automation 

TAC  is  highly  automated  in  Aircraft  Maintenance  scheduling,  but  Civil 
Engineering  and  Supply  are  essentially  manual  in  scheduling. 
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Figure  8  reflects  the  degree  of  scheduling  automation  in  the  "Other" 
MAJCOMs  (essentially  USAFE  and  PACAF). 


AUTO  AUTO/MAN  MAN/  MAN 
IN  AUTO  IN 


Figure  8.  "Other"  Degree  of  Scheduling  Automation 

What  automation  there  is,  exists  in  Aircraft  Maintenance  scheduling,  and 
possibly  a  small  amount  in  Civil  Engineering. 

In  terms  of  scheduling  systems,  it  would  appear  appropriate  to  examine 
SACs  systems,  particularly  in  Aircraft  Maintenance.  Also  of  interest  would  be 
further  examination  of  SACs  Supply  scheduling  system. 

The  following  question  was  among  those  asked  of  Training  Managers  on  the 
survey: 

Q.  33:  Please  use  the  following  scale  to  rate  each  aspect  of  your 
documentation  activity.  (Among  the  aspects  listed):  Integrating 
documentation  with  other  training  functions,  such  as  scheduling  and 
reporting. 

a.  Not  a  problem  at  present  -  no  improvement  needed. 

b.  Only  minor  problems  with  this  at  present. 

c  Some  problems  with  this  -  improvement  desirable. 

d.  Major  problems  with  this  -  improvement  urgently  needed. 

e.  Not  relevant  to  documentation. 

Over  40%  of  the  Training  Managers  in  Civil  Engineering  cited  problems  in 
integrating  documentation  with  other  training  functions.  Figure  9  examines  the 
Civil  Engineering  responses  to  question  33  by  MAJCOM,  and  graphically  portrays 
that  the  most  significant  problems  are  perceived  by  Training  Managers  in  SAC.  In 
fact,  no  other  command  sees  major  problems  with  this  activity,  and  MAC  sees 
very  little  problem  with  it. 
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Figure  9.  Civil  Engineering  Training  Manager  Responses  to  Question  33 

It  is  recommended  that  SAC  Civil  Engineering  functional  managers  be 
queried  as  to  their  perceptions  of  problems  with  documentation  integration  in  an 
attempt  to  rectify  the  situation.  This  may  be  directly  related  to  the  low  degree  of 
automation  found  in  the  Civil  Engineering  training  system;  however,  it  is 
somewhat  unusual  that  one  MAJCOM  would  be  so  predominant  in  identifying 
integration  as  a  problem. 


Trainer  Significant  Findings  Regarding  Degree  of  Automation 

The  following  question  was  among  those  asked  of  Trainers  on  the  survey: 

Q.  31:  Please  use  this  set  of  responses  to  indicate  how  frequently  you  use  each 
of  the  following  methods  when  delivering  training: 

a.  Use  this  method  with  nearly  all  trainees 

b.  Use  method  with  about  half  of  trainees 

c  Use  method  with  about  a  quarter  of  trainees 

d.  Have  used  this  method,  but  rarely 

e.  Never  used  this  method 
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Figure  10  reflects  selected  Trainers'  responses  to  frequency  of  use  of  Computer- 
assisted  instruction  (CA1). 


ALL  HALF  QTR  RARE  NEVER 
Figure  10.  Use  of  CAI 

As  can  be  seen,  little  use  of  CAI  is  reported. 

The  following  question  was  among  those  asked  of  Trainers  on  the  survey: 

Q.  57:  When  technologies  for  training  delivery  (e.g.,  automated  systems)  are 
introduced  in  our  unit,  they  are  used  to  full  capacity. 

a.  Strongly  agree 

b.  Agree 

c  Disagree 

d.  Strongly  disagree 

e.  Not  applicable  or  don't  know 

Figure  11  portrays  the  responses  of  selected  Trainers. 
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Figure  11.  Selected  Trainer  Responses  to  Question  57 


26 


Figure  12  collapses  the  responses  into  the  generic  "Agree”  or  "Disagree" 
categories  to  get  a  clearer  view. 
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Figure  12.  Collapsed  Responses  to  Trainer  Question  57 

SAC,  by  far,  shows  the  strongest  agreement,  and  TAC  the  strongest 
disagreement  that  training  technologies  are  used  to  full  capacity  in  their  units. 

The  following  question  was  among  those  asked  of  Trainers  on  the  survey: 

Q.  59:  Computer-assisted  training  technologies  can  be  as  effective  as 
classroom  training. 

a.  Strongly  agree 

b.  Agree 

c  Disagree 

d.  Strongly  disagree 

e.  Not  applicable  or  don't  know 

Figure  13  portrays  the  responses  of  selected  Trainers. 
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Figure  13.  Selected  Trainer  Responses  to  Question  59 
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Figure  14  collapses  the  responses  into  the  generic  "Agree”  or  "Disagree" 
categories  to  get  a  clearer  view. 
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Figure  14.  Collapsed  Responses  to  Trainer  Question  59 

SAC,  once  again,  shows  the  strongest  agreement,  with  other  commands 
showing  about  equal  disagreement. 

It  seems  clear  that  Trainers  in  SAC  are  the  strongest  supporters  of  computer- 
assisted  instruction  among  the  large  MAJCOMs. 


Trainee  Significant  Findings  Regarding  Degree  of  Automation 

The  following  question  was  among  those  asked  of  Trainees  on  the  survey: 

Q.  21:  For  each  method  of  training  listed,  please  use  these  responses  to 
indicate  whether  you  would  prefer  more  or  less  of  this  method  of  training  in 
the  future. 

a.  Prefer  much  less  of  this  method  of  training 

b.  Prefer  slightly  less  of  this  method  of  training 

c.  Present  use  of  this  method  is  about  right 

d.  Prefer  slightly  more  of  this  method  of  training 

e.  Prefer  much  more  of  this  method  of  training 

f.  Did  not  experience  this  method,  or  do  not  know 
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Figure  15  reflects  selected  Trainees'  responses  to  individual  training  using 
Computer-based  instruction  (CBI), 
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Figure  15.  Selected  Trainee  Responses  to  Question  21 

Figure  16  collapses  the  responses  into  the  generic  "More"  or  "Less"  categories 
to  get  a  clearer  view. 


Figure  16.  Collapsed  Responses  to  Trainee  Question  21 


Of  those  who  express  an  awareness  of  CBI,  there  is  a  uniform  indication  that 
more  CBI  use  is  desired  by  Trainees. 
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Time  Spent  on  Training  Duties  by  Training  Managers  versus  Degree  of 
Automation 

The  following  questions  were  among  those  asked  of  Training  Managers  on 
the  survey: 

Q.  9:  About  what  portion  of  your  duty  time  is  spent  on  scheduling? 

a.  All  or  nearly  all 

b.  About  three-quarters 

c  About  half 

d.  About  one-quarter 

e.  About  5  to  10  percent 

f.  Less  than  5  percent 

Q.  10:  How  would  you  describe  the  systems  you  currently  use  to  do  your 
scheduling? 

a.  Fully  automated 

b.  Automated  scheduling  from  manual  input 

c.  Manual  scheduling  from  automated  input  (source  documents) 

cl  Fully  manual 

Figure  17  plots  question  9  against  question  10. 
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Figure  17.  Time  Spent  Scheduling  vs.  Degree  of  Automation 

These  data  indicate  that  those  Training  Managers  using  automated 
scheduling  methods  tend  to  spend  more  time  on  the  activity  of  scheduling  than 
do  those  using  manual  methods.  Since  this  is  counterintuitive,  further 
investigation  was  undertaken. 
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When  looking  at  time  spent  on  scheduling  by  types  of  units,  we  fir  d  that 
high  time  is  reported  by  Training  Managers  in  Aircraft  Maintenance,  Medical, 
Services,  Training  Units,  and  Other.  When  looking  at  degree  of  automation  in 
scheduling  systems  by  types  of  units,  we  find  a  relatively  high  degree  of 
automation  reported  by  Training  Managers  in  Aircraft  Maintenance, 
Communications,  and  Training  Units.  A  relatively  low  degree  of  scheduling 
automation  is  reported  by  Training  Managers  in  Civil  Engineering. 

It  would  appear  that  Aircraft  Maintenance  and  Training  Units  are  the  largest 
contributors  to  the  "high  time/high  automation"  phenomenon  for  scheduling. 

The  following  questions  were  among  those  asked  of  Training  Managers  on 
the  survey: 

Q.  25:  About  what  portion  of  your  time  is  normally  spent  on 

documentation? 

a.  All  or  nearly  all 

b.  About  three-quarters 

c.  About  half 

d.  About  one-quarter 

e.  About  5  to  10  percent 

f.  Less  than  5  percent 

Q.  26:  How  would  you  describe  the  system  you  currently  use  to  document 

completed  training? 

a.  Fully  automated 

b.  Partially  automated 

c  Fully  manual 

Figure  18  plots  question  25  against  question  26. 


Figure  18.  Time  Spent  Documenting  vs.  Degree  of  Automation 
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These  data  also  indicate  that  those  Training  Managers  using  automated 
documentation  methods  tend  to  spend  more  time  on  the  activity  of  documenting 
than  do  those  using  manual  methods.  Again,  since  this  is  counterintuitive, 
further  investigation  was  undertaken. 

When  looking  at  time  spent  on  documentation  by  types  of  units,  we  find  that 
high  time  is  reported  by  Training  Managers  in  Training  Units,  Communications, 
Civil  Engineering,  and  Other.  When  looking  at  degree  of  automation  in 
documentation  systems  by  types  of  units,  we  find  a  relatively  high  degree  of 
automation  reported  by  Training  Managers  in  Training  Units,  Communications, 
and  Other.  A  relatively  low  degree  of  automation  is  reported  by  Training 
Managers  in  Civil  Engineering. 

It  would  appear  that  Training  Units,  Communications,  and  Other  are  the 
largest  contributors  to  the  "high  lime/high  automation"  phenomenon  for 
documentation . 

We  next  examined  time  spent  and  degree  of  automation  in  documentation  by 
MAJCOM.  (Tests  of  significance  relative  to  scheduling  did  not  make  examination 
of  that  aspect  by  MAJCOM  appropriate.) 

Figure  19  portrays  documentation  times  reported  by  Training  Managers  from 
MAC,  SAC,  TAC,  and  Other. 
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Figure  19.  Time  Spent  by  Training  Managers  in  Documentation 


Note  the  lesser  amount  of  time  reported  by  SAC  Training  Managers  in 
documentation. 
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We  then  examined  documentation  methods  reported  by  Training  Managers 
by  MAJCOM.  The  "big  four"  data  are  shown  in  Figure  3-20. 
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Figure  20.  Degree  of  Automation  in  Documentation  Methods 

Note  the  high  use  of  partial  automation  reported  by  SAC  for  documentation. 

Based  on  the  conclusion  that  SAC  appears  to  be  relatively  efficient  in  the  use 
of  at  least  partial  automation  (low  time/high  automation),  it  is  suggested  that  their 
methods  be  studied  in  more  detail  for  possible  use  as  a  "technology  baseline." 
Further  investigation  is  also  warranted  of  those  "high  time/High  automation" 
activities  identified  earlier. 


Problems  Identified  by  Trainers 

In  questions  11  to  20,  respondents  were  given  various  aspects  of  OJT  training 
and  were  asked  to  use  the  following  scale  to  rate  each  aspect: 

a.  Not  a  problem  at  present  —  no  improvement  needed. 

b.  Only  minor  problems  with  this  at  present. 

c  Some  problems  with  this  -  improvement  desirable, 

d.  Major  problems  with  this  —  improvement  urgently  needed. 
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Figure  21  reflects  the  distribution  of  responses  from  all  Trainers  responding  to 
the  survey.  (Refer  to  Appendix  B-2  for  descriptions  of  each  aspect  of  OJT  training 
addressed.) 
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Figure  21.  Magnitude  of  Problems 

Figure  22  collapses  the  responses  from  the  previous  figure  into  two  responses 
for  each  question  —  "No  and  Minimum  Problems"  into  one,  and  "Some  and 
Major  Problems"  into  another. 
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Figure  22.  Problems /No  Problems 

As  can  be  seen,  more  problems  are  reported  with  questions  15, 19,  and  20  than 
others.  These  questions  are: 

Q.  15:  Having  enough  time  to  perform  my  training  duties. 

Q.  19:  Training  on  tasks  that  are  not  performed  very  often. 

Q.  20:  OJT  keeping  up  with  changes  in  technology  used  for  the  job. 
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Figure  23  portrays  the  percentages  of  the  "Major  Problems"  responses. 
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Figure  23.  Major  Problems 

Again,  we  see  that  most  major  problems  are  reported  to  be  with  questions  15, 
19,  and  20. 

We  then  examined  each  of  these  questions  by  responses  received  from 
Trainers  in  MAC,  SAC,  TAC,  and  "Other"  (essentially  USAFE  and  PACAF). 

Figure  24  portrays  the  responses  to  question  15  (Having  enough  time  to 
perform  my  training  duties).  Chart  1  reflects  all  responses,  and  Chart  2  collapses 
the  responses  into  two  responses  —  "No  and  Minimum  Problems”  as  one,  and 
"Some  and  Major  Problems"  as  another. 


Qml  Chart! 


Figure  24.  Selected  MAJCOM  Responses  to  Question  15 

As  can  be  seen  the  magnitude  of  problems  with  question  15  varies  little  by 
MAJCOM. 


35 


MAC  SAC  TAC  OTHER 


MIC  SAC  TAC  OTHER 


Figure  25.  Selected  MAJCOM  Responses  to  Question  19 

It  would  appear  that  MAC,  especially,  experiences  some  problems  with 
training  on  tasks  that  are  not  performed  very  often. 

Figure  26  portrays  the  responses  to  question  20  (OJT  keeping  up  with  changes 
in  technology  used  for  the  job).  Chart  1  reflects  all  responses,  and  Chart  2  collapses 
the  responses  into  two  responses  -  "No  and  Minimum  Problems"  as  one,  and 
"Some  and  Major  Problems"  as  another. 
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Figure  26.  Selected  MAJCOM  Responses  to  Question  20 

As  can  be  seen,  the  magnitude  of  problems  with  question  20  varies  little  by 
MAJCOM. 
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We  then  examined  the  percentages  of  Trainers  in  selected  MAJCOMs  who 
reported  that  major  problems  exist  with  aspects  of  training  addressed  in  questions 
15, 19,  and  20.  Figure  27  portrays  those  responses. 
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Figure  27.  Percent  of  Trainers  Who  Reported  Major  Problems  by  MAJCOM 

Overall,  indications  of  the  existence  of  major  problems  are  minimal,  and 
their  magnitudes  vary  by  MAJCOM.  In  general,  overseas  respondents  ("Other") 
seem  to  perceive  more  major  problems  than  do  CONUS  respondents. 

We  then  examined  the  responses  provided  to  questions  15, 19,  and  20  by 
Trainers  in  selected  types  of  units,  specifically  Aircraft  Maintenance,  Civil 
Engineering,  Communications,  Security  Police,  and  Supply  units. 

Figure  28  reflects  responses  to  question  15  from  these  Trainers.  Chart  2 
collapses  the  responses  from  Chart  1  into  two  responses  for  question  15  —  "No  and 
Minimum  Problems"  into  one,  and  "Some  and  Major  Problems"  into  another. 
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Figure  28.  Selected  Unit  Responses  to  Question  15 
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Figure  30.  Selected  Unit  Responses  to  Question  20 
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As  can  be  seen,  the  magnitude  of  problems  with  OJTs  ability  to  keep  up  with 
changes  in  technology  does  not  vary  greatly  by  type  of  unit,  although  Civil 
Engineering  experiences  more  problems  than  others. 

We  then  examined  the  percentages  of  Trainers  in  selected  units  who  reported 
that  maior  problems  exist  with  aspects  of  training  addressed  in  questions  15, 19, 
and  20.  Figure  31  portrays  those  responses. 
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Figure  31.  Percent  of  Trainers  Who  Reported  Major  Problems  by  Unit  Type 

Overall,  indications  of  the  existence  of  major  problems  are  minimal,  and 
their  magnitudes  vary  by  type  of  unit  Reasons  for  perceived  major  problems  by 
Trainers  in  Civil  Engineering  regarding  question  20,  and  Trainers  in  Supply 
regarding  question  19  appear  worthy  of  further  examination. 


Interview  Significant  Findings 

Copies  of  the  interview  instruments  used  for  the  telephone  interviews  may 
be  found  at  Appendix  C.  Following  are  the  significant  findings  regarding  training 
documentation,  training  methods,  and  potential  for  expanded  unit  level  training. 


Significant  Findings  Regarding  Training  Documentation 

The  most  frequently  mentioned  problems  with  training  documentation  were: 

•  Keeping  documentation  current  and  accurate.  (Mentioned  by  36.4% 
of  the  Training  Managers,  and  17.5%  of  the  Trainers.) 

•  Takes  too  much  time.  (Mentioned  by  24.2%  of  the  Training 
Managers,  and  20%  of  the  Trainers.) 

•  It  is  paperwork-intensive.  (Mentioned  by  18.2%  of  the  Training 
Managers,  and  20%  of  the  Trainers.) 
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These  problems,  taken  in  combination,  indicate  that  training  documentation 
has  great  potential  for  further  automation. 

Significant  Findings  Regarding  Training  Methods 

When  asked  what  training  methods  are  used  in  their  unit,  there  was 
considerable  agreement  between  Training  Managers,  Trainers,  and  Trainees  with 
regard  to  the  use  of  self-study  materials,  operational  equipment,  classroom 
instruction,  videotapes,  and  simulators/mock-ups.  However,  as  can  be  seen  in 
Table  4,  there  was  considerable  disagreement  in  perceptions  as  to  the  use  of 
computer-based  training  (CBT),  non-operational  equipment,  and  interactive  video 
disk  (IVD). 


Table  4.  Training  Methods  Used  in  Unit 


Method 

Tmg  Mgr 

Trainer 

Trainee 

CBT 

55.3% 

17.5% 

40.0% 

Non-operational  Equipment  42.1% 

37.5% 

87.5% 

IVD 

17.5% 

17.5% 

7.5% 

Training  Managers  and  Trainers  were  asked,  "Are  there  any  training 
technologies  that  are  not  available  to  you  that  would  benefit  your  training 
program  or  activities?”  Forty-five  percent  of  Training  Managers  and  35  percent  of 
the  Trainers  responded  in  the  affirmative.  Of  those,  the  percentages  shown  in 
Table  5  specified  the  dted  technologies. 


Table  5.  Desired  Technologies 


Position 

CBT 

Training  Manager 

70.6% 

47.1% 

Trainer 

35.7% 

42.9% 

This  is  a  fairly  strong  statement  of  support  and  desire  for  expanded  use  of 
training  technologies,  especially  for  CBT. 
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Significant  Findings  Regarding  Potential  for  Expanded  Unit  Level  Training. 

Although  concerns  were  expressed  regarding  any  proposed  reduction  in  ATC 
resident  training,  63%  of  the  Training  Managers  and  60%  of  the  Trainers 
interviewed  indicated  that  training  levels  could  be  increased  at  the  unit  level 
without  a  negative  impact  on  operational  capability.  Their  opinions  varied 
considerably  as  to  the  magnitude  of  the  capacity  to  expand;  however,  most 
responses  clustered  between  25  and  50  percent 


Interviews  of  Desert  Stonn/Desert  Shield  Participants 

All  interviewees  were  asked  if  they  participated  in  either  Desert  Storm  or 
Desert  Shield,  or  both.  The  research  team  hoped  to  obtain  useful  data  regarding 
training  in  a  wartime  environment.  Unfortunately,  the  very  small  population 
interviewed,  particularly  those  who  deployed  to  the  Middle  East  (see  Table  6),  did 
not  allow  the  team  to  draw  any  meaningful  conclusions. 


Table  6.  Desert  Shield /Storm  Participants 


Tmg  Mgrs  Trainers  Trainees  Total 


Total  Participants 

12 

18 

2 

32 

Number  Who  Deployed 

1 

15 

0 

16 

Deployed  to  Middle  East 

1 

9 

0 

10 

Summary  of  Significant  Survey  and  Interview  Findings 

Only  30  percent  of  Trainers  are  instructor-qualified. 

Use  of  training  technologies  is  limited  in  most  MAJCOMs  and  units. 

In  terms  of  use  and  support  of  automated  training  systems  in  general. 
Training  Managers  and  Trainers  in  what  was  SAC  far  out-pace  the 
other  MAJCOMs. 

By  type  of  unit,  it  appears  that  Aircraft  Maintenance  is  the  most 
automated  in  its  training  systems,  and  Civil  Engineering  the  least. 

More  use  of  computer-assisted  instruction  is  desired  by  Training 
Managers,  Trainers,  and  Trainees. 
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•  Training  Managers  who  use  automated  scheduling  and  documentation 
systems  tend  to  report  more  time  spent  on  these  activities  than  do 
Training  Managers  using  manual  systems.  This  phenomenon  could 
not  be  fully  explained  through  analysis  of  data  collected.  It  occurs 
largely  in  Aircraft  Maintenance,  Communications,  and  Training  Units. 
In  documentation,  particularly,  it  may  be  that  those  with  automated 
systems  feel  "forced"  to  use  them,  whereas,  since  AFR  50-23  (Enlisted 
Specialty  Training)  does  not  "dictate"  much  in  the  way  of 
documentation  requirements,  those  without  an  automated 
documentation  system  do  not  spend  as  much  time  performing  this 
activity. 

•  The  most  significant  problems  identified  by  Trainers  are: 

•  •  Not  having  enough  time  to  perform  training  duties. 

••  Conducting  training  on  tasks  that  are  not  performed  very  often. 

••  OJTs  inability  to  keep  up  with  changes  in  technology  used  for 
the  job. 

Possible  solutions  to  these  problems/issues  will  be  addressed  in  the  next 
section  of  this  report. 
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IDENTIFICATION  OF  PROJECTS  TO  ADDRESS  SLT  NEEDS 


Current  Projects 

Many  of  the  research  efforts  currently  being  conducted  or  planned  by  AL/HR 
address  various  aspects  of  the  most  significant  concerns  and  needs  identified 
through  the  surveys  and  interviews.  These  needs  should  be  addressed  in  order  to 
enhance  or  improve  squadron  level  training  capabilities.  These  were: 

•  Only  30%  of  OJT  trainers  are  instructor  qualified. 

•  Time  spent  on  documentation  and  scheduling  is  excessive. 

•  Need  to  improve  the  currency  of  training. 

•  Need  to  increase  the  amount  of  time  OJT  trainers  have  to  train. 

•  Need  methods  for  addressing  training  on  tasks  seldom  performed. 

The  following  research  efforts  were  reviewed  to  determine  which  could  have 
some  impact  in  addressing  the  concerns  and  needs  listed  above. 


Table  7.  On-going  Research  Efforts 


Advanced  Instructional 

Computer-Based  Training 

Job  Performance 

Design  Advisor  (AIDA) 

Selection  and 
Implementation  Strategies 

Measurement  (JPM) 

Advanced  On-the-job 

Logistics  Command  and 

Training  System  (AOTS) 

Fundamental  Skills 

Training  (FST) 

Control 

Advanced  Training 

Multi-Task  Trainer 

System  (ATS) 

Instructional  Support 

System  (ISS) 

Research  (MTTR) 

Aircrew  Combat  Mission 
Enhancement  (ACME) 

Integrated  Maintenance 

Part  Task  Trainers  (PIT) 

Information  System 

Task  Analysis  for 

Basic  Job  Skills 

(IMIS) 

Tactical  C2  (TATC2) 

C-130  Aircrew  Training 

Intelligent  Tutoring 

Training  Assessment 

System  Evaluation  (ATSE) 

Systems  (ITS) 

Technologies 

Comprehensive 

Job- Aiding/Training 

Training  Decision  Modeling 

Occupational  Data 

Analysis  Technologies 

Technologies  (TDMT) 

Analysis  Program 
(CODAP) 

(JATAT) 

Training  System  for 
Maintenance  (TRANSFORM) 
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Using  the  ISD  Framework 


Of  the  projects  reviewed,  those  identified  as  having  some  direct  relevance  to 
the  SLT  needs  identified  were  then  compared  against  ISD  categories  using  the 
methodology  developed  in  Phase  I  of  the  SLT  project.  The  majority  of 
classifications  given  were  done  by  HRTC  and  HRTI  of  the  Armstrong  Laboratory, 
at  the  request  of  HRTE.  Table  9  at  the  end  of  this  section  gives  the  areas  of  the  ISD 
framework  addressed  by  the  research  projects. 

Using  the  ISD  classifications  as  a  guide,  the  selected  projects  were  analyzed  to 
determine  what  these  projects  could  contribute  to  squadron  level  training.  The 
projects  analyzed  did  not  have  to  specifically  address  or  focus  on  squadron  level 
training,  rather  projects  were  considered  relevant  if  they  provided  a  methodology, 
product  or  any  findings  that  could  be  applied  to  SLT  needs. 


Current  Research  with  Applications  to  SLT 

The  following  are  brief  descriptions  of  the  salient  (SL  related)  aspects  of  the 
research  efforts  with  comments  on  how  they  may  impact  squadron  level  training. 

Advanced  Instructional  . Design  Advisor  .(AIDA).  This  research  is  exploring  the 
automation  of  an  instructional  design  system  which  would  guide  course 
developers  through  the  instructional  design  process.  The  system  would  be 
derived  from  empirically  sound  and  theory-based  instructional  models. 

Comment.  The  automation  of  instructional  design  could  favorably  impact 
the  currency  of  training.  If  the  approach  is  very  user  friendly  it  could  provide 
assistance  to  squadron  level  training  development  by  providing  guidance  in 
materials  development. 

Base  Training  System  (BTS).  This  program  will  integrate  training  development, 
delivery,  evaluation,  and  management  to  provide  an  automated  squadron-level 
technology  for  systematic  and  effective  job-site  training.  The  production  version 
of  this  technology  is  currently  focused  on  management  functions. 

Comment.  This  technology  could  help  address  identified  concerns  regarding 
the  currency  of  training,  time  available  for  training,  and  issues  of  tasks 
seldom  performed.  The  implementation  of  the  BTS  part  of  this  project  may 
help  reduce  the  documentation  problem. 

Advanced  Training  System  (ATS).  This  CBT  development  and  support  package  is 
an  integrated  CBT  and  support  system  using  distributed  architecture  for 
development,  delivery,  evaluation,  and  management  of  training  at  ATC  resident 
technical  training  wings.  ATS  has  the  potential  to  be  the  foundation  for  all  CBT, 
simulators,  and  part-task  trainers. 
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Comm,  nt.  The  instructional  development,  delivery,  management,  and 
evaluation  subsystems  developed  under  this  effort  could  be  applied  to  SLT. 
One  factor  that  has  to  be  considered  is  the  current  lack  of  skills  at  the 
squadron  level  in  the  development  of  CBT,  simulators,  or  part-task  trainer 
development.  The  development  aspects  of  this  system  would  be  limited 
because  of  the  lack  of  skills  at  the  squadron  level  in  training  development. 

Basic  lob  Skills.  This  project  is  developing  an  integrated  cognitive  task 
analysis/training  development  technology  to  build  adaptive  training  directed  at 
the  fundamental  mental  skills  required  in  the  AFs  most  technologically  advanced 
enlisted  occupations.  The  goal  is  to  train  the  complex  problem  solving  skills  that 
accelerate  the  growth  to  competence  and  to  knowledge  flexibility  in  high-tech 
work  centers  by  capturing  the  collective  maintenance  knowledge  of  expert 
technicians.  The  expert  knowledge  base  is  then  transformed  into  leamable 
curricula  for  novice  technicians. 

Comment.  This  expert  system  could  be  applied  to  the  development  of  expert 
systems  based  programs  to  train  OJT  instructors,  provide  basic  instruction  and 
thereby  allow  OJT  trainers  to  focus  on  the  more  technically  difficult  tasks,  and 
also  provide  the  "shell"  for  job  aids  for  tasks  seldom  performed. 

Comprehensive  Occupational  Data  Analysis  Program  ICQD AP).  This  package  of 
computer  programs  is  used  to  input,  process,  organize,  and  report  occupational 
data  from  job  inventories.  CODAP  manipulates  and  reports  task-level  and 
biographical  survey  data  gathered  from  job  incumbents  and  expert  raters  for  the 
purpose  of  identifying  and  analyzing  current  job  structures  and  task  characteristics 
within  a  target  occupational  area. 

Comment  This  system  can  also  be  used  to  collect  and  analyze  data  on  OJT 
instructor  needs,  help  determine  if  training  is  current  by  comparing  CODAP 
findings  against  training  materials,  and  identify  those  tasks  that  are  seldom 
performed. 

Computer-Based  Training  Selection  and  Implementation  Strategies.  This  effort 
will  develop  guidelines  for  selecting  and  applying  CBT  technologies  to  training. 
The  products  will  help  guide  the  selection  of  existing  courses  for  conversion  to 
CBT. 


Comment.  At  the  squadron  level,  such  guidelines  could  help  Training 
Managers  and  Trainers  identify  information  that  can  be  effectively  presented 
through  CBT.  The  information  can  then  be  presented  to  developers. 

Instructional  Support-System  (ISS).  This  project  will  produce  a  user  friendly 
computer-based  training  and  management  system  that  can  be  used  by  Trainers  at 
the  squadron  level  to  develop  CBT  for  their  own  use  and  shared  with  other  units. 
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Comment.  The  development  of  a  user  friendly  authoring  system  may  have 
only  minimal  effect  at  the  squadron  level  due  to  the  lack  of  training 
development  training  or  experience. 

Integrated  Maintenance  Information  System  (IMIS).  This  effort  will  produce  a  tool 
to  provide  in-depth  training  for  maintenance  technicians  through  a  portable 
digital  text  and  graphics  display  job  aid.  This  electronic  job  aid  can  be  hand-carried 
to  present  the  required  information  at  the  job  site. 

Comment.  The  IMIS  can  have  a  direct  impact  on  tasks  seldom  performed 
and  overall  training.  Though  the  portability  aspect  may  not  be  essential  for 
many  non-maintenance  jobs,  the  concepts  and  structure  of  the  programming 
may  well  prove  beneficial  to  the  development  of  training  tools  and  job  aids 
outside  of  the  maintenance  career  fields. 

Intelligent  Tutoring  Systems  (ITS).  This  effort  is  intended  to  provide  effective,  low 
cost,  transportable  training  for  individualized  instruction  in  specific  high 
technology  areas.  ITS  authoring  and  delivery  capabilities  will  result  from  the 
Intelligent  Computer  Assisted  Training  Testbeds  (ICATT)  project. 

Comment.  The  use  of  artificial  intelligence  based  systems  for  the 
development  of  adaptive  training  based  systems  would  appear  to  be  a  natural 
outgrowth  of  the  use  of  AI  based  programs  for  training  development.  It  may 
well  enhance  the  quality  of  CBT  by  incorporating  adaptive  training 
techniques.  No  theoretical  reason  exists  for  not  using  the  ITS  for  less 
technical  occupations.  Training  would  need  to  be  developed  centrally  and 
then  provided  to  the  units. 

Tob-Aiding/Training  Analysis  Technologies  (IATAT).  This  project  will  provide 
methods  for  deciding  whether  and  how  tasks  should  be  trained,  job-aided,  or  some 
combination  of  the  two,  and  provide  human  and  system  performance  models  to 
evaluate  those  decisions. 

Comment.  This  could  be  used  at  the  squadron  level  to  systematize  the 
training  process,  and  thereby  allow  Trainers  to  focus  on  more  technically 
difficult  tasks. 

lob  Performance  Measurement  (TPM).  This  program  is  developing  methods  for 
measuring  individual  job  performance  to  validate  selection/classification  tests, 
evaluate  training  systems /programs,  and  evaluate  research  products. 

Comment.  In  order  to  assure  that  training  at  the  squadron  level  is  achieving 
its  desired  goal,  measurement  of  job  performance  is  critical.  The  evaluation 
of  training  against  actual  job  performance  is  a  much  more  desirable  and 
accurate  indicator  of  training  effectiveness  than  the  use  of  some  intermediate 
factor  of  training  effectiveness  such  as  end  of  course  assessments. 
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7/1  Ml 


Multi-Task  Trainer  Research  t 


SI,  This  effort  is  based  on  Air  Intercept  Trainer 


technology  and  will  provide  emergency  procedure  training  in  flight  safety.  It  will 
have  audioring  capabilities  so  that  non-programmers  can  quickly  and  easily 
modify  course  content  to  provide  up-to-date  training. 


Comment.  Though  the  content  of  the  training  will  be  useful  for  a  segment  of 
the  squadron  level  population,  the  approach  used  for  ease  of  updating 
content  could  well  influence  the  design  of  other  C8T  efforts  if  it  proves  to  be 
easily  transportable  and  not  related  to  characteristics  of  the  subject  matter. 


Part  Task  Trainers  (PTTs).  The  PTT  provides  low  cost,  off-the-shelf,  easily 
upgradeable  simulator  training  systems.  PTTs  would  permit  frequent  practice  of 
mission  critical  tasks  at  the  squadron  level. 


Comment.  The  reduction  of  cost  and  improving  the  ability  of  part  task 
trainers  to  be  upgraded  could  expand  the  use  of  simulator  training  systems 
into  areas  where  cost  has  been  prohibitive  and  where  rapid  changes  in 
technology  have  made  simulators  impractical.  Reduction  in  cost  of  PTTs 
would  permit  the  Air  Force  to  increase  the  number  and  availability  of 
simulators  at  the  base  and  squadron  level  which  could  in  turn  reduce  the 
amount  of  time  personnel  spend  at  formal  technical  schools.  PTTs  with  the 
characteristics  of  low  initial  cost  and  ease  of  upgrade  have  the  potential  of 
becoming  integral  to  squadron  level  training. 

Training  Impact  Decision  System  (TIDES).  The  findings  of  this  effort  will  allow 

Training  Managers  to  make  better  decisions  about  the  what,  when,  and  where  of 

training.  Decisions  could  be  made  that  focus  on  squadron  level  training  needs. 

Comment.  TIDES  is  primarily  a  high  level  decision  support  system  for  use  by 
functional  managers  of  career  fields.  It  is  possible  that  future  applications 
could  focus  on  squadron  level  training  issues.  This  could  prove  a  useful  tool 
by  giving  Training  Managers  better  information  on  the  decisions  they  need  to 
make. 


Training  System  for  Maintenance  (TRANSFORM).  The  system  developed  will  be 
used  by  Instructional  Systems  Developers  in  developing  maintenance  training  for 
new  weapon  systems.  The  TRANSFORM  products  will  be  used  by  Field  Training 
Detachments.  The  system,  which  provides  an  automated  interface  between  the 
Logistics  Support  Analysis  process  and  Instructional  Systems  Development,  is 
being  used  as  a  prototype  in  a  Joint  Service  Decision  Support  System. 

Comment.  This  system  could  well  have  a  positive  impact  on  the  currency  of 
training  at  the  squadron  level.  Consideration  should  be  given  to  expanding 
the  application  of  this  technology  to  other  than  weapons  systems. 
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Additional  SLT  Focused  Research 


The  needs  identified  as  the  highest  priorities  by  the  SLT  research  diverge 
somewhat  from  the  areas  of  focus  of  the  current  projects  at  Armstrong 
Laboratories.  The  following  section  deals  with  the  identification  of  research  efforts 
that  specifically  address  the  major  issues  identified  in  the  surveys  and  interviews 
conducted. 


ISSUE  ONE*  Only  30%  of  PIT  trainers  are  instructor  qualified. 

The  number  of  OJT  instructors  that  are  instructor  qualified  was  surprisingly 
low.  The  concern  here  is  not  a  question  of  credentials,  rather  it  is  a  matter  of 
training  knowledge.  OJT  trainers  may  well  possess  specific  technical  knowledge  in 
a  field,  yet  know  nothing  about  how  to  train.  Without  some  form  of  training,  OJT 
trainers  must  of  necessity  fall  back  on  what  they  remember  of  how  they  were 
trained  or  use  the  "discovery  method,"  that  is  figuring  it  out  for  themselves.  This 
may  or  may  not  lead  to  successful  training. 

OJT  is  the  foundation  of  squadron  level  training.  And  the  OJT  instructor  is 
the  single  most  important  factor  in  the  delivery  of  quality  training  at  the  squadron 
level.  Although  the  importance  of  die  OJT  instructor  is  recognized  as  critical,  for 
the  vast  majority  of  OJT  instructors,  instruction  is  a  secondary  task. 

Given  the  realities  of  the  work  environment,  a  training  method  needs  to  be: 

•  Available  on  the  job  site  on  demand. 

•  Able  to  provide  what  the  trainer  needs  to  know  in  a  manner  that  is 
understandable  and  effective. 

•  Flexible  enough  to  fit  any  job  environment. 

•  Implementation  and  maintenance  costs  must  be  very  low. 

Several  methods  to  address  this  issue  were  considered: 

•  CBT  for  the  trainer. 

•  Use  of  distance  learning. 

•  Expert  Systems  training  guidance. 

•  Enhanced  expert/professional  support  (better  Training  Managers). 

Of  those  listed,  two  are  implementable  within  a  relatively  short  period  of 
time  and  at  low  cost.  These  are: 

•  CBT  for  the  trainer  (OJT-Trainer  CBT);  and 

•  Expert  System  training  guidance  (Expert  Trainer). 

Computer  Based  Training.  A  CBT  program  can  be  developed  at  relatively  low 
cost  using  technology  available  today.  Given  the  widespread  availability  of  DOS- 


48 


based  computers  throughout  the  Air  Force  for  use  as  platforms,  the  use  of  CBT  is 
desirable.  A  course  can  be  shipped  worldwide  on  a  few  disks  and  can  be 
maintained  at  substantially  less  cost  than  paper-based  training  materials. 

Much  of  the  current  research  presently  conducted  by  AL/HR  is  focused  on 
Computer  Based  Training.  The  Computer-Based  Training  Selection  and 
Implementation  Strategies.  Instructional  Support  System,  Iptelliggnt  Tutoring 
Systems.  Multi-Task  Trainer  Research,  and  Part  Task  Trainers,  all  take  advantage 
of  the  computer  environment  as  a  modality  of  training  delivery.  A  research  effort 
to  develop  a  CBT  program  for  OJT  instructor  training  could  take  significant 
advantage  of  existing  findings  and  expertise.  To  develop  a  low  cost  OJT-Trainer 
CBT  program  should  not  take  more  than  a  year  to  research  and  develop  a 
prototype. 

General  Background  and  Concepts  of  CBT.  Computer  based  instruction  takes 
many  forms:  text  with  graphics,  interactive  video  disk  (IVD),  compact-disc  based 
systems  (CD-ROM,  CD-I),  digital  video  interactive  (DVD  and  others.  But  no  matter 
how  sophisticated,  die  key  element  of  all  these  systems  is  that  they  take  advantage 
of  the  computer's  ability  to  manipulate  different  types  of  information  --  audio, 
video,  graphics,  animation,  text  —  in  an  interactive  learning  situation. 

The  differences  between  the  various  types  of  CBI  systems  lies  largely  in  their 
particular  selection  of  information  types,  e.g.,  some  systems  emphasize  sound 
capabilities,  others  focus  heavily  on  graphics.  Because  of  this  mixing  of  various 
forms  of  information,  these  systems  are  often  referred  to  as  interactive 
multimedia  systems.  Although  the  word  multimedia  is  a  vague  term  that  is  often 
misused,  it  is  effective  in  describing  the  direction  in  which  CBI  is  heading. 

As  multimedia  technologies  become  more  sophisticated,  and  CBI  developers 
anticipate  the  integration  of  video  into  their  presentations,  those  that  are  worth 
their  salt  have  already  begun  to  seriously  consider  the  audio  and  visual  aspects  of 
CBI.  The  increasing  capability  of  moderately  priced  computer  hardware  to 
effectively  handle  complex  graphic  representations  and  audio  output  will  allow 
developers  to  move  from  printed  text  type  descriptions  of  an  activity  to  a  more 
naturalistic  situation  where  the  student  can  listen  to  the  abstract  concepts  while 
viewing  the  concrete.  Unlike  classroom  lectures,  the  auditory  information  is  not 
lost  in  the  computer  environment,  but  is  as  permanent  as  any  text  presented  in  a 
program. 

Furthermore,  the  notion  of  interactivity  has  been  heralded  as  the  key  factor  in 
making  computer  based  instruction  an  effective  teaching  instrument.  However, 
there  are  some  fundamental  misconceptions  about  interactivity  that  have  resulted 
in  scores  of  weak  learning  programs. 

Before  designing  a  CBI  program,  the  question  that  must  be  asked  is:  "what 
will  be  lost  in  translating  this  content  material  from  a  classroom  environment  to 
an  individualized  system?"  Many  developers  responded  to  this  translation  by 
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creating  CBI  programs  that  both  prompted  the  user  with  commands  like  "PRESS 
ANY  KEY  TO  CONTINUE"  and  posed  multiple  choice  questions,  believing  that 
the  result  was  an  interactive  program. 

However,  the  true  nature  of  interactivity  is  much  deeper  than  this.  The 
interactive  relationship  between  student  and  teacher  in  a  classroom  environment 
is  intellectual.  This  type  of  interactivity  cannot  be  replicated  in  a  CBI  by  simply 
forcing  the  student  to  push  buttons  on  a  keyboard. 

The  true  interactive  relationship  of  a  classroom  environment  is  recreated 
only  by  implanting  this  interactivity  in  the  content  of  the  subject  matter. 

Expert  Systems.  An  Expert  System  consultation  using  existing  artificial 
intelligence  programming  languages  such  as  ProLog  or  Ml  that  will  run  on 
existing  platforms,  can  be  developed  relatively  quickly.  The  OJT  trainer  would  use 
such  a  system  in  a  consultative  manner,  accessing  expert  advice  when  needed. 

The  development  of  an  OJT-Trainer  CBT,  or  an  Expert  Trainer  will  require 
basic  research  to  identify  OJT  training  methodologies  that  are  easily  learned  and 
can  be  applied  quickly  by  non-professional  trainers.  The  Intelligent  Tutoring 
Systems  project  can  provide  an  existing  base  for  the  development  of  the  Expert 
Trainer.  This,  like  the  previously  discussed  OJT-Trainer  CBT,  could  be  a  relatively 
"fast  turnaround"  project.  The  major  technical  hurdles  would  be  content  related 
rather  than  the  mode  of  delivery.  Such  an  effort  may  take  from  twelve  to 
eighteen  months. 

Some  of  the  issues  that  need  to  be  addressed  are:  What  are  the  most  common 
individual  problems  of  learners?  What  methods  can  be  used  to  address  these 
problems?  How  much  does  the  OJT  trainer  need  to  know  in  any  one  area  (depth 
of  training)  in  order  to  be  effective? 

General  Background  and  Concepts  of  Expert  System.  Expert  Systems  are  a 
direct  result  of  research  in  the  field  of  artificial  intelligence.  They  can  help  solve 
complex  problems  that  previously  required  the  aid  of  a  human  expert  with  years 
of  experience.  And  they  can  do  it  with  the  speed  and  recall  that  only  a  computer 
provides. 

Expert  Systems  are  typically  designed  around  a  Knowledge  Base  of  FACTS 
and  RULES  relating  to  a  particular  task  or  application,  and  an  Inference  Engine 
that  performs  the  reasoning  process  to  solve  specific  problems  in  that  application 
area.  Once  an  expert  system  is  developed  and  fielded,  computer  users  can  engage 
in  question-and-answer  consultations  with  the  system  about  a  problem  or  factors 
pertinent  to  the  application. 

Typically  these  systems  have  a  natural  language  interface  which  makes  the 
knowledge  contained  in  these  programs  accessible  to  almost  anyone.  Expert 
Systems  make  knowledge  accessible  in  a  useful  form  even  when  the  human 
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expert  is  not  physically  present.  Expert  Systems  can  allow  a  novice  to  function  "as 
if"  he  or  she  were  an  expert  within  a  specific  domain. 


An  Expert  System  is  very  different  from  computer  programs  that  supply  strict 
factual  knowledge  such  as  the  word  processing  spell  checkers  or  a  simple  question 
and  answer  program.  An  Expert  System  models  the  problem  solving  approach  of 
an  expert.  An  expert  system  is  not  something  that  can  be  used  to  solve  new  or 
novel  problems.  But  most  problems  are  only  new  to  the  individual  facing  the 
problem.  To  experts,  most  problems  are  simply  variations  on  a  theme. 

Expert  Systems  model  experts,  they  do  not  replace  them.  An  Expert  System 
cannot  provide  information  beyond  the  scope  of  die  information  available  to  it. 
For  example,  an  Expert  System  designed  to  assist  in  troubleshooting  television  sets 
could  help  to  troubleshoot  a  radio,  but  an  Expert  System  designed  to  troubleshoot 
radios  would  be  of  no  use  troubleshooting  the  video  components  of  a  television. 

Expert  Systems  use  human  knowledge  and  experience  to  solve  problems  that 
could  otherwise  be  solved  by  an  expert  in  a  reasonable  length  of  time.  An 
end-user  who  is  knowledgeable,  but  not  an  expert  in  a  specific  area,  will  benefit 
from  the  system's  ability  to  perform  the  role  of  the  expert  By  using  Expert 
Systems  to  distribute  dedsion-making  and  problem-solving  expertise,  an 
organization  can  optimize  its  resources  and  reduce  its  costs.  Expert  Systems  can 
enhance  productivity  by  making  this  expertise  available  to  others,  helping  them 
make  decisions  and  solve  problems  effectively. 

Today's  most  successful  Expert  Systems  are  capable  of  solving  problems  in 
which  the  knowledge  needed  to  solve  that  problem  is  already  understood,  and 
which  can  be  solved  by  an  expert  in  a  reasonable  amount  of  time.  Such  problems 
are  typically  solved  by  an  expert  dividing  the  problem  into  several  sub-problems 
and  tackling  each  sub-problem  independently. 

These  problems  are  characterized  by  their  finite  set  of  possible  outcomes.  In 
the  problem-solving  process,  the  optimum  solution  is  determined  with  human 
reasoning  that  considers  the  evidence  for  and  against  possible  outcomes.  The  steps 
in  the  reasoning  process  involve  collating  and  abstracting  information  about  the 
problem,  relating  the  problem  to  a  general  class  of  solutions,  and  then  selecting 
and  defining  one  or  more  solutions. 

Other  Approaches  Considered.  Of  the  other  possible  approaches  considered, 
enhanced  professional  support  -  improving  the  skills  of  Training  Managers  in 
providing  support  to  the  OJT  Trainer  -  may  require  a  reconceptualization  of  the 
function  of  the  Training  Manager's  job.  Based  on  interview  findings,  many 
Training  Managers  view  themselves  primarily  as  schedulers  and  documenters  of 
training.  This  seems  partially  due  to  a  lack  of  training  knowledge,  and  to  the 
amount  of  time  many  spend  on  documentation  and  reporting.  Providing 
enhanced  professional  support  may  well  require  changes  in  policy  regarding  the 
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major  job  of  the  Training  Manager  and  improvement  of  the  documentation  and 
reporting  technology. 


ISSUE  TWO!  Time  spent  on  documentation  and  scheduling  is  excessiyfi. 

To  reduce  the  time  spent  on  documentation  without  reducing  quality  will 
require  an  attack  on  this  problem  from  several  directions. 

The  following  ways  of  addressing  the  problem  were  considered: 

•  CBT  user  training  and  improved  documentation. 

•  High  level  data  integration  of  data  sources. 

•  User  friendly  interface  and  on-line  help. 

•  Self-checking  integrated  systems. 

The  findings  described  in  this  report  indicating  that  Training  Managers  using 
automation  for  scheduling  and  documentation  spend  more  time  on  these 
activities  should  be  taken  with  some  concern.  The  direction  that  documentation 
and  scheduling  appears  to  be  moving  in  the  Air  Force  is  towards  increased 
automation  and  standardization.  Current  efforts  being  implemented  should 
increase  standardization.  The  most  immediately  implementable  means  to  address 
this  problem  would  be  providing  computer  based  training  on  the  automated 
documentation  and  scheduling  systems. 

Analysis  of  the  user  interface  to  determine  if  a  more  user  friendly  interface  is 
needed  and  the  development  of  user  friendly  interfaces  could  not  only  reduce 
training  time,  but  reduce  the  amount  of  time  spent  on  the  system.  An  on-line 
help  should  also  yield  relatively  large  gains  in  time  reduction  with  a  relatively 
modest  investment  of  research  time  and  money. 

Self-checking  integrated  systems,  which  would  require  both  improvement  in 
the  programming  and  a  high  degree  of  integration  among  various  data  sources, 
could  have  a  significant  impact  on  the  time  spent  on  scheduling  and 
documentation.  This  is  viewed  as  a  long-term  high  cost  effort.  Improving  data 
integration  and  developing  self-checking  systems  are  much  more  complex  tasks 
with  many  technical  hurdles  to  overcome. 

General  Concepts  and  Background  in  Information  Flow  Technology.  The  use 
of  computer  technology  to  manage  the  flow  of  information  in  an  organization  can 
yield  increased  efficiency.  Until  recently,  the  only  way  to  maintain  and  access  large 
volumes  of  information  was  through  die  use  of  mainframe  and  minicomputer 
technology.  This  was  often  done  at  great  expense  with  the  need  for  unique 
software  and  dedicated  machines.  But  today,  the  revolution  in  networking 
technologies  has  made  electronic  information  management  a  possibility  at  the 
microcomputer  level. 
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Networking  software  allows  any  number  and  type  of  computers  to  be  linked 
together.  At  the  next  level,  these  small  dusters  of  computers  can  then  in  turn  be 
linked  to  other  groups  of  computers  through  standard  telephone  or  other  cable 
wiring  systems. 

The  key  in  allowing  the  interchange  of  information  between  different 
machines  is  the  use  of  a  set  of  pre-ordained  protocols  for  the  conversion  of  data  to 
match  the  input  requirements  of  each  computer  in  the  chain.  These  interface  units 
use  data  exchange  standards  ordained  by  organizations  like  the  American  National 
Standards  Institute,  The  Consultative  Committee  on  International  Telephone  and 
Telegraph,  and  the  Government  Open  Systems  Interconnect  Profile  at  the 
National  Institute  of  Standards  and  Technology. 

The  benefit  of  the  electronic  management  of  data  is  threefold: 

1.  The  availability  and  access  of  information  is  increased. 

2.  The  information  can  be  updated  globally. 

3.  The  information  can  be  manipulated  and  analyzed  rapidly  and 
efficiently. 

The  management  materials  and  information  through  electronic  means  for 
Squadron  Level  Training  would  present  all  of  the  above  benefits,  the  most 
relevant  being  item  #2.  The  increased  ability  to  update  records  would  reduce 
redundancies,  time  spent  inputting  data,  and  increase  the  accuracy  of  the 
information. 

One  factor  that  needs  to  be  considered  is  APR  50-23  (Enlisted  Specialty 
Training).  From  the  standpoint  of  developing  scheduling  and  documentation 
tools  it  is  imperative  that  the  Air  Force  determine  what  information  is  required. 
AFR  50-23  provides  little  guidance. 


ISSUE  THREE:  Need  to  improve  the  currency  of  training. 

Often  the  response  to  research  that  addresses  an  "old"  problem  such  as 
currency  of  training  is  less  than  enthusiastic,  people  and  organizations  leam  to 
live  with  it  and  accept  it  as  a  "fact  of  life."  But  what  is  currently  a  continuing 
nuisance  rather  than  a  crisis  is  likely  to  become  a  serious  problem  in  the  future. 
The  rate  of  technological  change  is  increasing.  Change  is  occurring  at  a  much 
more  rapid  rate  today  than  at  any  time  in  the  past.  Assuming  that  no  change 
occurs  in  the  rate  of  change,  the  content  of  much  of  today’s  training  will  become 
obsolete  faster  than  at  any  time  in  the  past.  If  the  methods  of  maintaining 
currency  of  training  are  not  advanced  at  the  same  rate  as  that  of  other  areas  of 
technology,  it  will  become  increasingly  difficult  to  maintain  even  a  semblance  of 
currency.  Even  though  it  is  an  old  problem,  we  feel  that  it  should  be  addressed 
now  so  as  to  avoid  a  future  crisis  in  Air  Force  training. 
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This  is  a  complex  problem  with  no  technically  or  conceptually  simple 
solutions.  Before  selecting  any  methodology  to  address  this  problem,  the  entire 
process,  from  training  developed  in  conjunction  with  new  equipment  or 
upgrades,  to  delivery  of  training,  needs  to  be  studied.  The  identification  of 
bottlenecks,  whether  they  be  the  result  of  the  decision  making  process,  production, 
or  delivery,  must  be  identified.  This  is  an  essential  first  step  that  will  require 
studies  to  determine  how  things  should  be  done  as  specified  in  directives  and 
regulations,  and  how  things  are  done  as  determined  by  a  study  of  practice. 

The  following  are  possible  ways  of  addressing  the  problem  that  should  be 
considered  once  research  to  determine  major  factors  of  the  problem  is  completed: 

•  Automated  CBT  training  update. 

•  On-line  updates  and  supplemental. 

•  Provide  "Just  In  Time"  training. 

•  Use  embedded  training. 

Automated  CBT  Training  Update.  The  cost  effectiveness  of  this  approach  is 
dependent  on  the  sheer  quantity  of  CBT  training  available  at  the  squadron  level. 
Conceptually  the  process  is  very  simple,  updates  and  corrections  to  CBT  training 
programs  are  electronically  transferred  and  programs  at  the  receiving  end  are 
automatically  updated.  This  approach  to  updating  would  require  that  every 
site/base  have  a  central  computer  for  the  storage  of  programs.  Trainees  would 
then  be  given  programs  retrieved  from  storage  on  the  central  computer,  whether 
through  a  local  area  network,  or  simply  copied  onto  a  disk.  Given  the  high  level 
of  reliability  in  networking  and  communications  technology  and  the  relatively 
simple  task  of  designing  standardized  data  files  and  file  names,  all  that  is  required 
is  a  proof  of  concept  study  and  a  prototype  system.  The  most  difficult  part  of 
applying  such  a  system  is  imposing  a  standardized  system  for  file  naming  and 
requiring  its  use.  Developing  the  prototype  system  could  take  as  little  as  six 
months  of  development  time  for  two  researchers.  Convincing  Training  Managers 
or  decision  makers  of  the  worth  of  this  effort  may  be  more  difficult. 

On-line  Updates  and  Supplementals.  This  approach  is  similar  to  the 
Automated  CBT  Training  Update,  but  specialized  for  the  reception  and 
reproduction  of  printed  training  materials.  The  only  important  differences  are  the 
need  for  the  creation  of  document  structures  that  allow  for  easy  upgrade,  fast  laser 
printers,  and  reproduction  capability.  The  front  end  tasks  of  this  project  are  much 
more  difficult  than  the  information  transfer  and  delivery  aspects.  How  does  one 
get  training  materials  upgraded  and  updated  quickly,  and  with  no  loss  in  accuracy? 


lust  in  Time  Training.  "Just  In  Time"  training  is  a  style  of  training  that  is 
based  upon  two  fundamental  premises.  The  first  is  that  there  is  an  improved 
retention  of  information  and  training  for  a  particular  type  of  work  if  and  when  the 
training  is  received  as  close  as  possible  to  the  time  in  which  it  is  needed.  The 
second  is  that  training  efforts  are  streamlined  if  the  training  takes  place  only  when 
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the  skills  are  required.  The  development  of  a  Just  in  Time  training  structure 
requires  very  accurate  documentation,  training  and  career  path  planning,  and 
advanced  programs  to  review  records  and  determine  training  needs. 

Embedded  Training.  The  use  of  embedded  training  is  most  rewarding  in 
situations  where  the  task  is  performed  infrequently  or  where  accurate 
performance  is  critical.  Embedded  training  simply  involves  the  inclusion  of 
relevant  training  materials  in  the  system  or  set  of  tools  required  for  that  particular 
task. 


For  example,  a  computer  system  that  is  used  to  diagnose  the  cause  of  engine 
failure  in  any  given  machine  may  be  fairly  complex.  The  process  of  diagnosis 
would  require  a  "funnel"  of  decisions  beginning  from  a  broad  base  of  potential 
problems,  eventually  growing  narrower  until  the  cause  of  the  engine  failure  is 
found.  A  system  of  this  type  may  not  be  complex  in  its  design,  but  the  user  must 
have  the  advantage  of  understanding  the  subject  matter  well  enough  to  answer 
the  question  accurately. 

In  this  sort  of  situation,  a  training  program  embedded  in  the  tool  itself  would 
allow  a  user  to  access  the  required  banks  of  information  with  relative  ease.  This 
avoids  situations  where  a  user  is  forced  to  deal  with  a  computer  system  that  they 
are  not  familiar  with  or  do  not  have  the  necessary  knowledge  to  take  advantage  of. 
Furthermore,  it  is  often  awkward  for  users  to  admit  that  they  are  not  familiar  with 
a  given  system  or  task.  Embedded  training  assures  that  human  error  caused  by 
this  sort  of  situation  will  be  minimized.  Except  for  the  special  case  of  computer- 
based  programs  and  tools,  most  embedded  training  is  built  into  a  system.  Adding 
embedded  training  to  existing  equipment  is  usually  not  a  feasible  option. 


ISSUE  FOUR:  Increase  the  Amount  of  Time  PIT  Trainers  Have  to  Train. 

Time  to  train  is,  in  a  very  real  sense,  addressed  by  all  the  previous  methods 
discussed,  and  by  many  of  the  projects  currently  being  carried  out  by  the 
Armstrong  Laboratory.  Effective  use  of  time  is  an  overarching  concern.  Since  the 
transfer  of  knowledge  and  the  development  of  skills  takes  time,  the  trainers  and 
training  managers  are  most  concerned  in  training  new  personnel  in  an 
operational  setting  where  training,  of  necessity,  is  secondary  to  the  job.  The  more 
quickly  a  trainee  can  become  proficient  in  a  job,  the  faster  it  translates  into 
improved  readiness,  reduction  of  manpower  requirements,  and  overall  cost 
reduction. 

The  use  of  automated  job  aids  to  eliminate  training  clearly  reduces  training 
time.  Automating  training  through  the  use  of  CBT  or  expert  systems  for  common 
or  simple  tasks  will  allow  the  Trainer  to  spend  time  with  the  Trainee  focusing  on 
the  more  conceptually  and  technically  difficult  tasks. 
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When  a  task  is  seldom  performed,  maintaining  proficiency  is  near  impossible 
unless  heroic  training  efforts  are  put  forth.  This  is  done  very  often  in  the  Air 
Force.  Emergency  first  aid,  weapons  training,  mission  training,  and  safety  training 
are  some  of  the  tasks  that,  though  seldom  performed,  are  continuously  trained 
and  retrained.  Pilots  for  example,  are  continuously  trained  to  fight  battles  that,  we 
hope,  will  never  be  fought.  Yet,  maintaining  proficiency  is  an  important  aspect  of 
deterrence. 

For  other  tasks,  continuous  training  and  retraining  is  not  a  realistic  option. 
But  without  training,  proficiency  of  a  skill,  whether  it  be  an  administrative  task 
seldom  performed  or  maintenance  of  highly  reliable  equipment,  will  decay  over 
time.  Some  current  and  proposed  research  already  discussed  can  contribute 
significantly  to  addressing  this  problem.  Today,  portability  and  reliability  of 
computer  based  job  aids  are  not  serious  hurdles.  Current  AL/HRT  research  is 
addressing  many  of  the  issues  involved  in  developing  job  aids  for  low  frequency 
tasks,  but  some  research  is  needed  in  the  following  areas: 

•  Standardization  of  job  aid  formats  for  expert  systems. 

•  Effectiveness  and  efficiency  of  generalists  training  in  a  job  aid  rich 
environment. 

The  use  of  expert  system  based  job  aids  is  a  feasible  solution  for  low  frequency 
tasks.  Such  job  aids  can  be  incorporated  into  existing  systems  or  be  developed  for 
unique  portable  platforms.  They  should  prove  effective  if: 

•  The  user  knows  how  to  use  the  job  aid,  i.e.,  there  is  training  provided 
for  its  use. 

•  The  user  has  general  job  knowledge  and  technical  skills  in  order  to 
apply  expert  system  direction. 

•  The  job  aid  itself  is  current. 

Standardized.  Format-  Automated  job  aids  are  very  useful  but  could  be 
confusing  to  the  user  if  every  job  aid  has  its  own  screen  layout  and  command 
structure.  The  development  of  a  standardized  layout  for  all  job  aids  will  reduce 
the  problems  caused  by  idiosyncratic  control  functions,  and  screen  layout. 

Generalists  Training.  The  arguments  between  the  proponents  of  the 
technical  generalists  versus  those  of  the  specialists  will  not  be  solved  in  any 
research  proposed  here.  The  issue  to  be  discussed  her  is  whether  it  would  be 
advantageous  for  the  Air  Force  to  train  generalists  in  a  job  aid  rich  environment. 

It  is  generally  agreed  that  generalists  give  up  depth  while  specialists  give  up 
breadth,  given  equivalent  training.  With  an  increasing  use  of  job  aids  it  may  be 
possible  for  the  generalists  to  perform  specific  tasks  with  the  same  facility  as 
specialists  and  also  perform  a  broader  range  of  tasks  than  any  one  specialist. 


56 


At  this  point  what  is  needed  is  the  development  or  application  of  a 
methodology  for  determining  the  skills  and  knowledge  required  of  generalists  as 
compared  to  specialists.  AL/HR  has  conducted  studies  using  methodologies  that 
would  be  appropriate.  A  study  conducted  by  Lamb  et.  al.  (Dec  1987)  used  cluster 
analysis  to  group  tasks  in  a  study  of  small  unit  maintenance  specialists  for  the  F-16. 
The  identification  of  knowledge  and  skills  required  of  generalists  cannot  be  bound 
by  current  groupings  of  AFSCs  and  job  families,  even  though  these  groupings  may 
prove  to  be  the  most  appropriate.  Possibly  the  single  most  difficult  aspect  of  this 
research  effort  is  having  no  preconceptions  regarding  the  findings. 


Research  Opportunities 

The  research  topics  previously  discussed  focused  on  addressing  current 
training  needs.  But  AL/HRT  should  not  limit  itself  to  an  essentially  reactive 
approach.  Some  new  technologies  and  improvements  in  older  technologies  are 
opening  up  new  areas  of  training  research  which  could  have  a  revolutionary  effect 
on  the  entire  training  structure,  including  the  capabilities  of  training  at  the 
squadron  level.  This  section  will  discuss  potential  research  areas  which  may 
produce  exciting  results. 


Virtual  Reality 

Virtual  Reality  (VR)  is  a  conceptual  approach  to  user  interaction  with 
"synthetic  environments".  Synthetic  environments  are  wide-angle, 
three-dimensional  stereo,  computer  generated  "realities",  that  are  visually 
convincing,  and  respond  in  real-time  to  the  users'  interactions.  This  technology  is 
fast  becoming  an  affordable  alternative  for  military  education  and  training. 

VR  as  a  mode  of  instructional  delivery  for  training  has  two  clear  advantages: 

•  The  ability  to  simulate,  model  and  visualize  both  situations  and  data  in 
order  to  enhance  the  students'  abilities  to  analyze,  interpret,  and  make 
decisions: 

••  by  providing  a  visually  realistic  depiction  of  complex  data  or 

information  to  assess  relationships  and  patterns  that  would  not 
be  visible  in  an  uncontrolled  environment  or  with 
alphanumeric  data. 

••  by  allowing  a  user  to  view  a  simplified  version  of  an  otherwise 
complex  phenomenon. 

•  The  ability  to  create  environments  for  learning  which  not  only  allow 
students  to  apply  and  develop  skills  at  a  comparatively  low  cost,  but 
also  offers  the  possibility  of  accelerating  the  learning  process  itself  by 
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turning  the  students’  entire  perceptions  of  reality  into  a  fully 
functional  interactive  classroom. 


Some  important  research  currently  being  conducted  by  the  Air  Force  in  this 
area  includes: 

•  3-Dimensional  Audio  research  exploring  technology  that  can  project 
sound  in  a  three-dimensional  space  using  a  headset  and  the 
implications  of  human  abilities  to  localize  that  information. 

•  Low  Fidelity  Helmet  development  efforts  provide  a  low-cost 
alternative  to  high-cost  fiber  optic  helmets.  These  developments  can 
be  applied  to  virtual  world  applications. 

•  Virtual  Man-Machine  Interaction  projects  are  being  conducted  to 
explore  virtual  space  simulators.  Low-cost  virtual  worlds  can  be 
applied  to  training  novices  to  fix  technologically  complex  equipment  in 
a  virtual  environment. 

It  is  important  to  understand  that  simulation  and  virtual  reality  are  two 
related  but  separate  concepts.  Simulation  research  is  directed  towards  the  creation 
of  realistic  synthetic  environments,  primarily  for  training.  VR  technology,  on  the 
other  hand,  is  an  effort  to  create  a  set  of  interface  tools  that  allow  computer  users 
to  move  beyond  keyboards  and  monitors,  and  step  inside  these  computer¬ 
generated  synthetic  environments. 

Although  the  public  fascination  with  this  technology  is  a  relatively  recent 
phenomenon,  the  research  and  development  work  behind  VR  has  been  silently 
taking  place  in  government,  university,  and  private  sector  labs  for  more  than 
thirty  years. 

This  research  has  yielded  a  semi-standard  configuration  for  the  full 
immersion  VR  system.  This  configuration  consists  of  a  head  mounted  display 
(HMD),  an  interface  de-  ice  (most  commonly  a  glove  or  force-feedback  joystick),  a 
spatial  tracking  system  (to  translate  the  user's  movements  into  the 
computer-generated  environment),  and  a  workstation  class  computer  with 
appropriate  software. 

The  initial  development  of  VR  both  as  a  set  of  tools  and  as  a  concept  took  off 
in  the  late  1970s  and  the  early  1980s,  in  a  variety  of  places:  The  Advanced  Research 
Projects  Agency  (ARP A),  NASA/ Ames,  Atari's  Sunnyvale  Laboratory,  and  MITs 
Media  Lab,  to  name  a  few.  More  recently,  the  technology's  relative  maturity  has 
fueled  the  growth  of  a  new  breed  of  lean  yet  powerful  companies,  like  VPL 
Research,  Inc.  and  Silicon  Graphics,  that  are  aggressively  marketing  applications  of 
VR  technology  in  both  the  public  and  the  private  sector.  The  stunning  success  of 
these  companies  has  propelled  computer  heavyweights  like  Apple  Computer  and 


IBM  to  look  more  seriously  into  the  possibility  of  VR  as  a  mass  commercial 
technology. 


The  role  of  the  military  in  sponsoring  the  beginnings  of  VR  research  was  a 
crucial  factor  in  the  development  of  this  technology.  It  was  through  the 
sponsorship  of  both  ARPA  and  the  Office  of  Navi  Research  that  the  first 
experiments  with  head  mounted  displays  for  VR  were  conducted  by  Ivan 
Sutherland  in  1966,  first  at  MITs  Lincoln  Laboratory,  and  later  at  the  University  of 
Utah. 

The  sponsorship  of  NASA,  ARPA  and  other  military  agencies  created  a  fertile 
ground  for  many  of  the  breakthroughs  in  the  development  of  the  early  VR 
systems.  This  occurred  not  only  through  the  availability  of  funding,  but  also 
through  the  indoctrination  of  researchers.  Many  of  the  companies  that  now 
dominate  the  small  but  burgeoning  VR  industry  were  started  by  researchers 
coming  out  of  military  VR  and  simulation  labs. 

If  adapted,  virtual  reality  will  cause  two  major  shifts  in  the  teaching/leaming 
process,  neither  of  which  has  a  solid  base  of  research  from  which  to  develop  actual 
software.  First,  curriculum  will  no  longer  be  print  based,  but  instead  will  be  based 
on  imagery  (both  auditory  and  visual).  Second,  learning  will  shift  from  the  use  of 
printed  abstractions  to  the  use  of  simulations  or  experiential  re-creations  of  events 
in  which  students  participate. 

Much  has  to  be  learned  about  the  characteristics  of  this  technology  for  the 
purpose  of  instruction.  A  suggested  research  effort  is  one  that  will  focus  on  the 
identification  of  critical  design  and  development  parameters  related  to  the  use  of 
virtual  reality  (VR)  for  training.  To  achieve  the  goal,  two  tasks  need  to  be 
accomplished: 

•  The  analysis  of  current  capabilities  of  virtual  reality  hardware  and 
software. 

•  The  identification  and  definition  of  critical  parameters  that  influence 
the  effectiveness  of  the  virtual  reality  environment  for  learning,  skills 
building  and  transfer  of  training. 


Distance  Learning 

Distance  learning  is  a  need-based  phenomenon  —  it  arises  out  of  the  need  to 
make  information,  and  more  importantly  education,  accessible  and  cost-effective. 
This  need  is  more  pressing  in  today's  Air  Force  than  ever  before.  While  new 
distance  learning  techniques  such  as  video  conferencing,  digital  video  and  cable 
TV  narrowcasting  are  being  developed  and  applied,  the  more  traditional  methods 
that  have  been  used  in  the  past  have  yet  to  be  abandoned. 
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In  the  past,  the  needs  that  stimulated  the  development  of  distance  learning 
were  primarily  physical  —  issues  of  geography,  culture  and  mobility.  These 
obstacles  forced  educators  to  evolve  their  strategy  so  they  could  continue  to  fulfill 
their  teaching  mission.  The  result  was  the  creation  and  eventual  acceptance  of 
distance  learning  as  a  legitimate  form  of  instruction.  The  earliest  such  application 
of  distance  learning  can  be  traced  to  the  1880s  when  William  Briggs  established  the 
somewhat  grandly  named  "University  Correspondence  College"  in  England  to 
make  higher  education  available  on  a  wide  basis. 

The  same  logistical  problems  exist  in  today's  Air  Force.  The  strong  emphasis 
on  cost  and  personnel  reductions  has  created  an  even  more  pressing  need  for  the 
re-discovery  and  re-development  of  innovative  training  techniques  such  as 
distance  learning.  This  new  urgency  is  also  supplemented  by  the  rapid  growth  in 
the  volume  of  information  and  the  diversity  of  skills  that  must  be  learned  by  Air 
Force  personnel  to  allow  them  to  manage  the  situations  they  must  encounter  on 
the  job. 

Today  in  the  United  States,  correspondence,  radio,  and  television  are 
commonly  used  for  distance  learning.  In  addition  to  these  "traditional" 
technologies,  new  technologies,  such  as  Computer-Based  Instruction  and 
videotaped  courseware  have  already  had  a  significant  impact  on  distance  learning. 

The  AT&T  Long  Distance  Learning  Network  (LDLN),  is  a  good  example  of  the 
impact  of  technology  on  distance  learning.  LDLN  is  a  large  network  which 
supports  "learning  circles"  in  which  five  to  eight  classrooms  (from  several 
counties)  are  grouped  together  to  cooperate  on  common  projects.  Teachers  and 
students  can  send  individual  messages  through  a  computer  terminal,  and  post 
messages  to  a  shared  electronic  "folder". 

It  would  not  be  an  overstatement  to  say  that  every  advance  made  in 
communications  and  information  transfer  technologies  has  eventually  been 
applied  to  distance  learning  -  and  there  is  no  reason  to  believe  that  this  process  of 
adding  technologies  to  the  distance  learning  repertoire  will  not  continue. 
Consequently  the  identification  of  the  most  appropriate  technologies  to  use  for  any 
course  or  set  of  courses  is  becoming  much  more  complex.  In  order  to  provide  the 
best  quality  distance  learning  at  the  least  cost,  it  is  essential  to  keep  current  with 
ever  changing  technologies.  The  "one  hammer"  approach,  where,  regardless  of 
the  training  requirements,  one  technology  is  always  used  would  only  guarantee 
the  production  and  delivery  of  mediocre  training. 

From  the  standpoint  of  this  study,  current  research  addressing  questions 
related  to  distance  learning  should  be  expanded  to  consider  other  modes  of 
information  delivery  with  a  view  toward  minimizing  costs  and  maximizing 
effectiveness. 

The  increasing  availability  of  different  types  of  communications  technology 
has  resulted  in  hybrid  technologies  which  are  very  powerful,  and  when  properly 
applied,  reduce  costs. 
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Video  teletraining,  audio  teletraining,  audiographic  teletraining,  and 
computer  conferencing  are  based  on  relatively  old  technology  —  conference  calls, 
modem  links,  and  television  signals  over  satellite  or  landline.  But  used  together, 
these  systems  are  able  to  re-create  —  with  varying  degrees  of  fidelity  —  a  classroom 
environment. 


An  Issue  for  the  Research  Community 

Occasionally  in  a  research  effort  some  findings  appear  which,  while  not 
directly  impacting  on  the  research  goal,  are  of  such  importance  that  not  discussing 
and  addressing  the  issue  would  be  a  disservice  to  the  sponsoring  organization  and 
the  integrity  of  the  research.  The  following  issue  is  of  such  a  nature. 

•  Need  to  increase  the  applications  of  training  technologies  as  they  apply 
to  squadron  level  training. 

Research  on  training  in  the  Air  Force  is  focusing  on  issues  directly  related  to 
the  current  and  future  needs  of  the  service.  As  such,  even  theoretical  research  can 
be  viewed  as  having  an  applied  base.  Success  of  a  research  effort  is  largely 
determined  by  its  effect  on  training  in  the  Air  Force.  How  to  maximize  the 
utilization  of  the  products  of  research  is  an  important  and  complex  issue.  Many 
factors  influence  whether  research  findings  "sit  on  the  shelf’  or  are  accepted  and 
implemented  by  the  Air  Force  community.  An  almost  certain  way  of  assuring  that 
research  findings  will  not  be  used  is  having  users  see  no  need  for  the  research,  or 
reject  the  approach  as  too  expensive  or  impractical.  This  study  has  shown  the 
relatively  small  influence  that  training  technology  appears  to  have  had  at  the 
operational  unit  level. 

In  both  the  surveys  and  interviews.  Trainers  and  Trainees  were  asked  what 
training  methods/tools  were  used  in  their  units.  Table  8  reflects  the  percentages  of 
respondents  who  indicated  that  they  had  experienced  the  use  of  the  dted 
technologies  in  their  units. 


Table  8.  Use  of  Training  Technology 


Trainers 

Trainees  ! 

Survev 

Interview 

Simulators /Mock-ups 

47.5% 

53.2% 

40.0% 

18.3% 

Computer-based  Training /Instruction 

17.5% 

36.9% 

40.0% 

24.7% 

Interactive  Video  disk 

17.5% 

21.2% 

7.5% 

9.2% 
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Although  Trainers  have,  at  some  time,  used  the  technologies  as  indicated 
-bove,  it  should  be  noted  that  their  use  (based  on  survey  results)  has  been 
minimal  at  best.  They  report  using  only  the  dted  technologies  for  50%  or  more  of 
their  Trainees  as  follows:  25.2%  simulators /mock-ups;  12.4%  CBT;  and  4.9%  IVD. 

This  is  considered  a  serious  problem.  Using  advanced  technology  to  provide 
training,  regardless  of  its  quality  and  effectiveness  as  an  instructional  tool,  is  no 
guarantee  that  the  training  will  be  used. 

In  order  to  systematically  depict  a  strategy  to  address  the  need  to  increase  the 
application  of  training  technology  to  SLT,  we  have  developed  a  Science  and 
Technology  Checklist  which  in  part  can  assist  in  overcoming  potential  barriers 
early  in  the  research  effort.  The  checklist  should  not  be  viewed  as  a  strategy,  but 
rather  as  strategy  development  guidance.  It  is  suggested  that  this  checklist  be  used 
as  part  of  the  initial  design  for  a  research  effort  and  that  certain  questions  be 
addressed  prior  to  proceeding  with  a  research  effort. 


Background  of  the  S&T  Checklist 

The  S&T  Checklist  can  serve  both  as  a  planning  tool  and  as  a  management 
tool.  As  a  planning  tool  it  can  be  used  to  identify  what  needs  to  be  done.  As  a 
management  tool  it  can  be  used  to  identify  what  is  being  done  as  well  as  what 
needs  to  be  done.  The  checklist  addresses  the  following  topic  areas: 


Analysis 

Cost  Benefit  Analysis 
Resource  Planning 
Test  and  Evaluation 
Documentation 


Coordination/Interface 
Priority  Determination 
Supportability 

Implementation/Deployment  Planning 
Follow-up 


Because  the  checklist  is  designed  to  be  used  for  all  training  technology 
research,  some  topic  areas  and  questions  may  not  be  applicable  to  a  specific 
research  topic. 

The  use  of  the  S&T  Checklist  as  a  tool  will  provide  the  following  advantages 
to  the  Air  Force. 

•  The  procedure  will  allow  the  investment  strategy  to  be  refined  over 
time,  even  during  the  actual  research  activity. 

•  The  same  procedures  can  be  applied  to  future  studies. 

•  It  provides  a  clear,  easily  followed  "paper  trail"  of  activities  leading  up 
to  the  decision  making  stage. 

•  It  reduces  the  possibility  of  important  factors  not  being  considered. 
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•  It  creates  a  common  framework  for  making  investment  decisions. 

•  It  provides  clear  guidance  for  the  development  of  an  S&T  investment 
strategy,  including  risks. 

The  S&T  Checklist  and  its  reference  document  are  tools  intended  for  long 
term  use  by  the  Human  Resources  Directorate  of  the  Armstrong  Laboratory  to 
make  decisions  on  what  specific  areas  of  research  should  be  investigated. 

The  checklist  is  intended  to  be  used  by  a  development  team.  By  using  the 
checklist,  information  can  be  shared  more  efficiently  and  the  current  status  of 
various  activities  can  be  documented.  For  example,  if  funding  is  not  available, 
then  the  status  of  funding  reflected  in  the  checklist  would  be  coded  RED.  Funding 
would  need  to  be  identified  in  order  to  proceed. 

The  S&T  Checklist  will  give  project  managers  and  other  decision  makers 
needed  documentation  and  history  which  may  assist  in  addressing  problems.  The 
checklist  will  provide  planners  and  decision  makers  with  a  tool  that  will  assist 
them  in  determining  if  a  particular  line  of  research  is  still  viable,  or  if  changes  in 
the  approach  are  needed. 

The  S&T  Checklist  does  not  replace  the  various  reports  and  documents 
needed  or  required,  it  supplements  them. 

Besides  its  use  as  a  planning  tool,  the  checklist  can  also  serve  as  a 
management  tool.  Through  a  process  of  periodic  reviews  and  updating,  a  line  of 
research  can  be  kept  on  track  end  problems  addressed  as  they  are  noted.  The  S&T 
Checklist  Guide  and  Checklist  can  be  found  at  Appendices  H  and  I  respectively. 


Science  &  Technology  Investment  Strategy  Checklist  Reference  Document 

To  assist  future  users  of  the  S&T  Checklist,  a  reference  document  was 
prepared  which  will  give  future  users  a  general  understanding  of  the  topics  to  be 
addressed  in  the  checklist  and  possible  strategies  to  address  each  topic.  The 
reference  document  provides  general  guidance  to  assist  users  to  identify  or 
develop  the  best  procedure  or  strategy  to  address  a  specific  checklist  item.  The 
reference  document  can  be  found  at  Appendix  J. 

Overall  current  research  efforts  address  various  aspects  of  the  squadron 
training  needs  identified.  Though  additional  research  efforts  and  enhancements 
of  current  research  are  proposed  in  this  report,  it  appears  that  what  is  needed  to 
maximize  the  benefits  (and  impact)  of  current  and  hiture  research  efforts  to  the 
Air  Force  is  a  strategic  plan  that  will  provide  a  systematic  strategy  to  achieve 
overall  training  goals. 
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Denotes  that  a  project  adds  to  the  technology  or  knowledge  base  of  a  process 


THE  RESEARCH  AGENDA 


Organization  of  the  Section 

A  number  of  individual  research  efforts  have  been  suggested  in  this 
document.  This  section  provides  an  overall  research  agenda  which  we  feel  will 
address  the  most  significant  problems  and  needs  of  training  at  the  squadron  level. 
This  agenda  is  divided  into  three  parts:  1)  Research  efforts  that  can  be  done  in  a 
relatively  short  time  frame  and  be  rapidly  implemented.  The  short  term  research 
efforts  are  all  low  risk  efforts  based  on  well  understood  technology.  2)  Studies 
designed  to  gather  and  present  information  which  could  lead  to  applied  research 
efforts  and  policy  changes.  3)  Long-term  or  high  risk  research  efforts  are  those  that 
will  require  a  significant  amount  of  time  to  develop  and  where  the  underlying 
theory  and/or  applications  hardware  and  software  is  in  flux.  Long-term  efforts 
focus  primarily  on  potential  future  problems,  needs,  or  opportunities  that  could 
significantly  enhance  the  quality  or  change  the  nature  of  future  training  .  Long¬ 
term  efforts  are  typically  medium  or  high  risk  in  that  they  can  be  impacted  by 
numerous  factors,  such  as  continuation  of  funding,  maintaining  long  term 
support  for  the  effort,  improvements  in  competing  technologies,  changes  in 
priorities,  and  maintaining  the  needed  management  and  research  staff. 

Within  each  category,  the  ordering  of  the  research  activities  is  based  on  the 
following: 

1.  The  importance  of  a  project  as  it  relates  to  current  SLT  needs. 

2.  The  potential  for  rapid  development  and  implementation. 

3.  The  likelihood  of  the  research  to  be  carried  through  successfully. 

For  each  research  topic  the  following  information  is  given: 

Impact  -  What  effect  this  research  is  likely  to  have  on  squadron  level  training. 

Risk  -  (High/Medium/Low)  -  The  likelihood  of  the  research  to  be  carried  out 
successfully,  and  if  carried  out  to  be  implemented. 
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Short-Term  Research  Efforts 

CBT  for  the  framer  (OTT-Tramer  CBT) 

Impact:  Increased  quality  of  training. 

Risk:  Low 

The  implementation  of  this  program  should  increase  the  quality  of  OJT  by 
improving  the  training  knowledge  and  skills  of  the  OJT  trainer.  The  risk  is  low 
since  CBT  as  a  delivery  technology  is  well  understood  and  the  content  for  such  a 
program  can  be  largely  based  on  existing  materials  developed  for  instructor  led 
training. 

Assuming  that  only  limited  front  end  research  needs  to  be  conducted,  an 
OJT-Trainer  CBT  can  be  developed  in  less  than  a  year  using  2.5  professional  person 
years.  The  distribution  and  implementation  of  an  OJT-Trainer  CBT  would  have 
Air  Force  wide  impact,  effecting  all  OJT  training  at  a  very  low  cost  in  time,  money, 
and  personnel. 

CBT  training  on  documentation  and  scheduling  for  automated  systems 

Impact:  Free  up  training  time  for  Training  Managers  and  Trainees. 

Standardization  of  instruction. 

Risk:  Low 

The  use  of  a  CBT  training  program  on  documentation  and  scheduling  may 
help  reduce  the  time  spent  by  training  managers  on  this  task.  Such  a  program 
would  help  reduce  errors  that  can  occur  in  this  process.  The  risk  is  very  low. 

Us&r  frtendly  interface  and. on-line  help  for  automated  scheduling  and 
documentation 

Impact:  Improve  quality  of  scheduling  and  documentation  by  reducing  errors 

and  reduce  time  spent  on  these  tasks. 

Risk:  Medium  (some  problems  may  arise  due  to  integration  into  existing 

systems). 

Developing  the  most  user  friendly  interface  possible  and  useful  on-line  help 
can  improve  the  quality  of  scheduling  and  documentation  by  reducing  time 
required  for  inputting  data  and  reducing  errors.  The  process  for  developing  a  user 
friendly  interface  and  a  useful  help  menu  can  be  divided  into  two  general  steps: 

I)  gather  user  input  on  how  they  work  with  the  system  and  what  types  of  help 
information  is  needed,  and  2)  develop  and  incorporate  user  requirements  into  the 
interface  and  on-line  help.  The  most  difficult  problem  faced  by  the  researchers 
would  be  the  integration  of  the  interface  into  a  system. 
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Impact:  Improve  the  quality  of  training  delivery. 

Risk:  Low 

The  use  of  an  Expert  System  to  provide  advice  and  assistance  to  OJT  trainers 
can  help  enhance  the  quality  of  training  by  providing  expert  advice  on  training 
delivery  questions.  This  effort  must  first  focus  on  identifying  training  delivery 
problems  faced  hy  trainers.  To  gather  this  information  both  interviews  of  trainers 
and  trainees,  and  observations  of  the  training  process  across  a  variety  of  squadron 
level  training  environments  are  required.  Training  experts  would  then  analyze 
the  problems  identified  and  propose  approaches  to  address  the  problem.  Once 
these  steps  are  accomplished  an  expert  system  consultation  program  would  then 
be  developed  to  help  trainers  identify  solutions  to  specific  training  problems. 


Studies 


Standardization  of  formats  for  expert  system  based  job  aids 


Impact:  Reduce  confusion  and  reduce  need  to  learn  new  formats. 

Risk:  Low 


The  Air  Force,  in  the  near  future,  will  likely  be  using  a  greater  number  of 
"intelligent"  job  aids.  The  use  of  a  standard  interface  format  will  reduce  the  time 
required  to  learn  to  use  various  job  aids,  and  reduce  frustration  that  can  result 
from  different  jobs  aids  having  different  command  structures.  The  end  product  of 
this  research  effort  would  be  a  report  describing  the  format  initial  layout, 
command  structure  and  standard  feedback  structure  to  be  included  in  all  job  aids. 

Analysis  of  current  and  projected  future  capabilities  of  virtual  reality  hardware 

and  software 

Impact  Enhance  future  training 
Risk:  Low 

The  likely  increased  use  of  virtual  reality  (VR)  programs  for  training  will 
require  the  Air  Force  to  select  appropriate  hardware  and  software.  With  the 
increasing  number  of  tools  being  developed  for  use  in  the  development  and 
delivery  of  VR  programs,  a  study  of  the  characteristics  of  VR  hardware  and 
software  is  essential.  The  document  produced  can  be  used  by  developers  to  select 
appropriate  tools  for  the  task  at  hand.  The  report  would  be  based  on  information 
collected  from  VR  technology  vendors  through  interviews  and  in-depth  analysis 
of  reports  and  studies. 
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Impact  Lead  to  the  development  of  a  structure  similar  to  ISD  to  guide  VR 
applications  development. 

Risk:  Low 

The  purpose  of  this  study  is  to  develop  guidance  for  VR  training  applications 
developers  similar  to  ISD  documentation.  In  order  to  accomplish  this  it  is 
important  to  identify  critical  parameters  of  the  VR  environment  effecting 
learning,  skills  building  and  transfer  of  training.  Most  on  the  needed  information 
is  available  but  disbursed  throughout  the  VR  research  and  development 
community.  This  research  effort  will  gather  the  information  through  archival 
studies  and  interviews  and  site  visits  wilh  developers  at  selected  research 
laboratories  and  development  houses  involved  in  VR.  Once  the  data  is  gathered, 
it  will  be  structured  as  guidance  for  the  development  of  VR  training  applications. 

The  use  of  automated  yob  aids  to  eliminate  training 

Impact:  Expand  capacity  of  the  technician,  and  free  up  training  time. 

Risk:  Low 

The  study  would  focus  on  the  impact  of  the  use  of  automated  job  aids  for  use 
in  place  of  training.  This  study  would  focus  on  the  impact  of  automated  job  aids  to 
replace  task-specific  training.  The  replacement  of  training  with  automated  job  aids 
can  lead  to  a  reduction  of  training  and  an  expansion  of  the  capacity  of  technicians 
to  perform  a  wider  variety  of  tasks.  On  the  basis  of  this  study  die  Air  Force  may  be 
able  to  increase  the  use  of  automated  job  aids  and  reduce  task-specific  training. 

fcigh  level  data  integration  of  data-soiaces 

Impact  Can  provide  necessary  training  data  regardless  of  the  type  of  system 
used  for  initial  storage. 

Risk:  High 

The  integration  of  sources  of  training  data  such  that  data  can  be  shared  by  all 
system  users  would  be  an  ideal  situation.  The  implementation  of  the  entire 
Advanced  Training  System  package  could  well  address  this  need.  Because  of  the 
high  cost  involved  a  detailed  implementation  study  should  be  conducted  to 
determine  a  strategy  of  implementation. 


Impact:  Could  lead  to  the  restructuring  of  Air  Force  job  structure  and  training. 

Risk:  Low  (such  a  study  would  provide  useful  information,  but  could 

possibly  have  no  effect  on  training.) 

Because  of  the  down-sizing  of  the  Air  Force  and  increasing  budget  constraints, 
the  need  may  arise  for  the  complete  restructuring  of  Air  Force  training.  The  Air 
Force  may  continue  to  find  itself  in  the  position  of  doing  more  with  less 
manpower. 

This  major  study  would  consider  the  feasibility  of  providing  generalists 
training  in  place  of  specialist  training  in  a  job  aid  rich  environment.  The  impact 
of  the  acceptance  and  implementation  of  the  findings  of  this  study  (if  the  findings 
prove  that  generalists  can  perform  as  well  as  specialists  in  a  job  aid  rich 
environment)  would  be  significant.  This  could  result  in  the  complete  rethinking 
of  the  Air  Force  training  concept. 

Long-Terxn/High  Risk  Research  Efforts 


Impact:  Squadrons  will  be  able  to  access  training  not  locally  available  This  will 

affect  those  jobs  where  relatively  few  specialists  are  at  any  or.e  location. 
Risk:  Medium 

The  use  of  distance  learning  can  have  a  significant  impact  at  the  squadron 
level  by  providing  access  to  training  not  locally  available.  Distance  learning  would 
likely  impact  most  heavily  those  jobs  where  relatively  few  specialists  are  at  any 
one  location.  The  concept  of  distance  learning  is  well  developed.  The  effort 
would  be  the  creation  of  fully  flexible  prototype  networks  that  could  provide  the 
level  of  interactivity  needed  for  a  specific  training  to  effectively  carry  out  training. 
Cost  and  maintenance  of  such  a  system  would  be  major  considerations.  As  a  first 
step  a  feasibility  study  should  be  conducted. 


Impact:  Provide  on-site  experts. 

Risk:  High  (Cost  factors  involved  in  training  and  retraining  may  prove 

prohibitive). 

This  effort  would  consider  the  feasibility  of  restructuring  of  the  75XXX  career 
field  such  that  75XXXs  would  act  as  general  training  SMEs  at  the  base  and  squadron 
levels.  The  availability  of  training  SMEs  at  every  base  would  increase  the  capacity 
of  units  to  locally  develop  high  quality  training  materials  and  programs  and 
provide  training  advice  and  assistance  to  OJT  trainers.  The  effort  would  include 


identification  of  knowledge  and  skill  requirements,  the  development  of  a  program 
of  training  for  75XXXs,  the  training  of  a  select  group,  and  the  study  of  their 
performance  and  impact  at  bases  of  assignment.  This  effort  is  a  high  risk  effort 
requiring  strong  commitment  of  operational  commands.  The  cost  of  such  an 
effort  would  be  significant  and  may  prove  to  be  prohibitive. 

Automated  CBT  training  update 

Impact:  Provide  current  and  consistent  training  worldwide. 

Risk:  High  (The  effectiveness  of  such  a  program  is  dependent  on  high  levels 

of  integration  of  information  and  the  existence  of  significant  numbers 
of  CBT  training  programs). 

An  automated  CBT  training  update  system  where  CBT  programs  are  updated 
automatically  would  assure  current  and  consistent  training  worldwide.  The  risk 
in  the  development  of  such  a  system  is  high  since  the  cost  effectiveness  would  be 
dependent  on  the  number  of  CBT  training  programs  in  use.  Such  a  system  would 
require  a  significant  degree  of  standardization  in  the  structure  of  CBT  programs 
including  the  standardization  of  the  development  language  and  file  structure. 

This  approach  should  be  considered  when  CBT  training  has  become  a  large 
portion  of  Air  Force  training. 

The  system  would  function  only  if  there  is  a  centralized  control  of  CBT 
programs.  All  commands  would  need  to  work  through  a  central  office/location 
for  the  management  of  computer-based  training,  most  likely  under  the  control  of 
ATC.  A  high  level  of  commitment  and  support  would  be  essential  for  such  a 
system  to  function  effectively.  No  research  currently  exists  on  how  such  a  concept 
would  function.  The  development  team  would  require  a  number  of  different 
skills  for  the  development  of  such  a  system  such  as  data  management,  wide  area 
networks,  CBT  design,  and  CBT  programming. 

On-line  updates  .and  supplementals 

Impact:  Improve  the  currency'  of  training. 

Risk:  High  (Effort  will  require  massive  networking,  standardization,  and  the 

availability  of  relatively  powerful  microcomputers). 

As  with  Automated  CBT  training  updates,  this  effort  would  be  a  major  effort. 
Centralized  control  would  be  essential  along  with  maintaining  a  high  level  of 
standardization. 
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Provide  "lust  In  Time”  training 


Impact:  Significantly  improve  the  quality  of  lifetime  training  in  the  Air  Force. 

Risk:  High 

This  project  is  basic  research  in  the  Air  Force  environment  to  develop  a 
methodology  for  determining  the  skills  and  knowledge  required  generalists. 
Such  research  would  address  boundaries  between  AFSCs,  required  depth  and 
breadth,  transfer  of  training  issues,  and  the  selection  of  formalized  ways  of 
identifying  the  structure  of  knowledge  and  the  structure  of  the  disciplines  within  a 
grouping  of  specialties.  Such  a  research  effort  cannot  be  bound  by  current 
groupings  of  AFSCs  and  job  families,  even  though  these  groupings  may  prove  to 
be  the  most  appropriate.  As  such,  the  methodology  for  developing  a  methodology 
to  determine  skills  and  knowledge  required  by  a  generalists  in  a  job  aid  rich 
environment  must  be  fairly  rigid  so  that  the  findings  would  not  be  biased  by  the 
researcher's  preconceived  notions  of  what  the  findings  will  support. 

Significant  problems  need  to  be  researched  for  which  no  basic  work  of 
significance  exists:  how  to  identify  an  individuals  training  needs  on  a  continuous 
basis,  what  type  of  administrative  structure  is  required,  and  how  to  assure  that  the 
material  is  current,  are  just  some  of  the  issues  that  would  need  to  be  addressed. 

For  such  an  effort  a  feasibility  study  would  be  essential.  The  amount  of  time 
and  the  type  of  personnel  required  for  the  development  and  implementation  of  a 
Just  In  Time  training  would  be  identified  in  such  a  feasibility  study. 


Conclusion 

Today's  Air  Force  is  the  best  in  the  world.  The  quality  of  today's  airman  is  in 
many  respects  the  best  in  the  history  of  the  U.S.  Air  Force.  The  enlisted  force  today 
is  the  most  highly  educated  cadre  of  enlisted  personnel  that  has  ever  been  in 
uniform.  This  is  an  enviable  position.  These  airmen  come  with  all  the  needed 
basic  skills  required  to  take  advantage  of  the  training  provided.  Yet  this  should 
not  lead  to  complacency.  The  down-sizing  of  the  Air  Force,  the  changing  world 
situation,  demographic  changes  in  the  U.S.,  increasingly  complex  and  rapidly 
evolving  technology,  and  reduction  in  the  available  budget  mean  that  today's 
airman  must  do  more,  and  know  more.  If  the  Air  Force  is  to  continue  to  maintain 
its  technological  edge,  it  is  essential  that  today's  and  tomorrow's  airmen  be  the 
most  professional  and  technically  competent  of  any  cadre  in  the  history  of  the  Air 
Force. 
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Appendix  A 

Instructional  System  Development 
Process  and  Survey  Item  Match 
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INSTRUCTIONAL  SYSTEM  DEVELOPMENT  PROCESS 
AND  SURVEY  ITEM  MATCH 


Interpreting  the  Data: 


Each  section,  process,  and  subcategory  of  the  ISD  process  is  identified  by  a  specific  code.  The 
following  is  the  index  to  the  meanings  of  the  code  numbers. 


1 .  ANALYZE  SYSTEM  4.3 

REQUIREMENTS  4.3a 

4.3b 

1 . 1  Training  Needs  Assessment  4.3c 

1.1a  analyze  problem  4.4 

1.1b  identify  parameters 

1 . 2  Define/Analyze  Job  Performance  4 . 5 

Requirements  .  4.5a 

1.2a  develop  task  listing  4.5b 

1.2b  analyze  job  tasks  4.6 

1.2c  establish  target  population  characteristics  4.6a 

4.6b 


2  DEFINE  EDUCATION  &  TRAINING  4.7 
REQUIREMENTS 


2 . 1  Select  Tasks  Requiring  Training 

2.2  Determine  Student  Prerequisites  5.1 

2 . 3  Select  Appropriate  Training  Setting  5 . 2 

2.4  Forecast  Resource/Logistic  Requirements  5.2a 

5.2b 

3  DEVELOP  OBJECTIVES  &  TESTS  5.2c 

5. 2d 

3.1  Develop  Objectives  5.2e 

3.1a  develop  criterion  and  objectives 

3.1b  develop  job  performance  testing  standards  5 . 3 

3.2  Develop  Tests  5.3a 

5.3b 

4  PLAN,  DEVELOP  &  VALIDATE  5.3c 

INSTRUCTION 


4. 1  Plan  Sequence  of  Instruction 

4.2  Select  Instructional  Method 
4.2a  evaluate  alternative 

instructional  strategies 
4.2b  evaluate  alternative 
management  strategies 
4.2c  establish  course  design  strategy 


Select  Instructional  Media 
evaluate  candidate  media 
select  instructional  media 
develop  system  specifications 
Determine  Resource  and  Funding 
Requirements 

Develop  Instructional  Materials 
author  instructional  material 
produce  instructional  material 
Validate  Instructional  Materials 
review  courseware  prototype 
individual  &  small  group  tryouts 
Validate  Complete  System 

CONDUCT  &  EVALUATE 

Deliver  Instruction 
Support  Instruction 
schedule  students 
schedule  resources 
track  student  progress 
manage  training  requirements 
maintain  &  update 
instructional  system 
Evaluate  Instruction 
evaluate  student  performance 
evaluate  job  performance 
measure  utility  &  <'ost-benefit 
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Survey  items  axe  coded  as  follows: 

First  Digit .  2-  Training  Manager  Survey  Form 

3-  Trainers  Survey  Form 

4-  Trainees  Survey  Form 

Second  Digit . Survey  Item  Number 

Lower  Case  Letter... Specific  Response 

Each  question,  or  when  possible  response,  is  identified  to  specific  1SD  categories.  Items  may  be 
members  of  multiple  categories.  Survey  items  which  are  purely  demographic  are  not  included  in 
this  listing. 
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1 

1.1 

1.1a 

1.1b 

1.2 


1.2a 

1.2b 

1.2c 


2- 22,  2-23,  2-24,  2-35,  2-36,  2-37,  2-38,  2-49,  2-52,  2-53,  2-57a.  2-60,  3-23,  3-24, 

3- 25,  3-58,  3-61 


2- 22,  2-23,  2-24,  2-35,  2-36,  2-37,  2-38,  2-52,  2-53,  2-57b,  3-23,  3-24, 

3- 25,  3-61 


2-6j,  3-8,3-9,3-10 


2 

2 . 1  2-22,  2-23,  2-24,  2-33,  2-49,  2-57c,  2-23,  3-24,  3-25,  3-58 

2 . 2  2-22,  2-23,  2-57d,  2-63f,  2-64f,  2-65f,  2-66f,  3-23,  3-24,  3-25,  3-69f,  3-70f,  3-7 If, 
3-72f 

2.3  2-8b,  2-33.  2-57c 

2.4  2-8b,  2-11,  2-33,  2-57f,  2-59,  2-60,  2-65,  2-66,  2-68,  2-69,  3-13,  3-71,  3-72,  3-74, 
3-75 


3  2-20, 3-21 

3 . 1  2-22,  2-23,  2-24,  3-23,  3-24,  3-25 
3.1a  2-58a,  2-59a,  2-60a,  3-64a,  3-65a,  3-66a 
3.1b  2-59c,  2-60c,  3-64c,  3-65c,  3-66c 

3 . 2  2-58b,  2-59b,  2-60b,  3-64b,  3-65b,  3-66b 

4  2-20,  2-21,  3-21,  3-22 

4 . 1  2-58c,  2-59d,  2-60d,  3-64d,  3-65d,  3-66d.  4-41 

4 . 2  2-58e,  2-59e,  2-60e,  3-64e,  3-65c,  3-66e,  4-39,  4-40.  4-4 1 .  4-42,  4-43,  4-44,  4-45, 

4-46,  4-47 

4.2a  2- 14c 

4.2b 

4.2c 

4 .3  2-55,  2-56,  2-58f,  2-59f,  2-o0f,  2-61,  2-62,  3-1 1,  3-63,  3-64f,  3-65f, 

3-66f,  3-67,  3-68,  3-87,  4-39,  4-40.  4-41,  4-42,  4-43.  4-44,  4-45,  4-46,  4-47 

4.3a 

4.3b 

4.3c  2-14a,  2-31 

4.4  2-6i.  2-31,  2-35,  2-36,  2-37,  2-38,  2-51.  2-55,  2-56,  2-65,  2-66,  3-12,  3-60,  3-63, 
3-71,  3-72 

4.5  2-6b,  2-22,  2-23,  2-24,  2.54,  2-55,  2-56,  2-58g,  2-59g,  2-60g,  3-11,  3-23,  24,  3-25, 
3-62,  3-63,  3-64g,  3-65g,  3-66g,  4-39.  4-40,  4-41,  4-42,  4-43,  4-44,  4-45,  4-46,  4-47 

4.5a 

4.5b 

4.5c 

4.6  2-22,  2-23,  2-24,  2-54,  2-58h,  2-59h,  2-60h,  3-23,  3-24,  3-25,  3-62, 

3- 64h,  3-65h,  3-66h 

4.6a 

4.6b 

4.7  2-6h,  2-22,  2-23.  2-24,  2-51,  3-23,  3-24.  3-25,  3-60,  3-64i,  3-65i.  3-66i, 

4- 7,  4-8,  4-9,4-10,  4-11,  4-12,  4-13,  4-14.  4-15.  4-16 
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5 

5 . 1  2-6c,  2-8c,  2-7,  2-49,  2-50.  2-52.  2-53,  2-54,  2-61,  2-62,  2-72,  2-74,  2-75.  2-76,  2-79. 
3-26,  3-27,  3-28,  3-29,  3-30,  3-31,  3-32,  3-33,  3-34,  3-35,  3-36,  3-37,  3-38,  3-39, 
3-40,  3-41,  3-42,  3-43,  3-44,  3-45.  3-46,  3-47,  3-48,  3-49,  3-50,  3-51,  3-52,  3-53, 

3- 58,  3-59,  3-61,  3-62,  3-67,  3-68,  3-80,  3-81,  3-88,  4-6,  4-17,  4-18,  4-19,  4-20, 

4- 21  4-22  4-23  4-24  4-25  4-28  4-32 

5 . 2  2-6d,’  2- 8a,'  2-6f,’  2-7,  2-25,2-26.  2-27,  2-28,  2-29,  2-30,  2-3 1 ,  2-32,2-33,  2-35.  2-36, 

2- 37,  2-38,  2-48,  2-60,  2-69,  2-77,  2-78,  3-55,  3-57,  3-82,  3-83,  3-84, 

3- 85,  3-86,  4-28.4-51 

5.2a  2-6a,  2-8d,  2-8e,  2-9,  2-10,  2-11,  2-12,  2-13a,  2-16,  2-17,  2-19,  2-34,  2-39,  2-63a. 

2- 64a,  2-65a,  2-66a,  3-16,  3-69a,  3-70a,  3-7 la,  3-72a,  4-31,  4-33,  4-38 

5.2b  2-9,  2-10,  2-11,  2-12,  2-I3b,  2-16,  2-17,  2-19,  2-63b,  2-64b,  2-65b,  2-66b.  2-68,  3-15, 

3- 69b.  3-709b,  3-7 lb,  3-72b,  3-74,  3-75,  4-35 

5.2c  2-6e,  2-8f,  2-9,  2-10,  2-1 1,  2-12,  2-13c,  2-15,  2-16,  2-17,  2-18,  2-19,  2-34,  2-39,  2-63c. 

2- 64c,  2-65c,  2-66c,  3-69c,  3-70c,  3-7 1c,  3-72c 

5. 2d  2-14d,  2-16,  2-17,  2-18,  2-19,  2-63d,  2-64d,  2-65d,  2-66d,  3-18.  3-44,  3-45,  3-46,  3-47. 

3- 48,  3-49,  3-50,  3-51,  3-69d,  3-70d,  3-71d,  3-72d,  4-29,  4-32,  4-34,  4-36.  4-37,  4-38 
5.2e  2-16,  2-19,  2-63e,  2-64e,  2  65e,  2-66e,  3-14,  3-17,  3-20,  3-44,  3-45,  3-45,  3-46,  3-47 

3-48,  3-49,  3-50,  3-51,  3-69e,  3-70e,  3-7  le.  3-72e 

5.3  2-30,  2-31,  2-32,  2-33,  2-35,  2-36,  2-37,  2-38.  2-40,  2-41,  2-42,  2-43.  2-44,  2-45.  2-46. 

2- 47,  2-65,  2-6,  2-80,  3-19,  3-54,  3-56,  3-71,  3-72,  4-52 

5.3a  2-6g,  2-67a,  2-67b,  2-68a,  2-68b,  2-69a,  2-69b,  3-10,  3-73a,  3-74a,  3-75a,  3-73b, 

3- 74b,  3-75b,  4-30 

5.3b  2-67c,  2-67d,  2-68c,  2-68d,  2-69c.  2-69d,  3-73c,  3-74c,  3-75c,  3-73d, 

3-74d,  3-75d 

5.3c  2-67c,  2-68e,  2-69e,  3-73e,  3-74e,  3-75e 
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Appendix  B 

Annotated  Survey  Forms, 
Phase  II 


(This  Appendix  contains  the  individual  surveys  administered  to  Training  Managers 
(B-l),  Trainers  (B-2),  and  Trainees  (3-3),  annotated  with  response  frequency 
percentages.  The  numbers  immediately  following  each  response  are  those 
percentages  of  respondents  who  selected  the  particular  response.  If  there  are  two 
numbers  following  a  response,  such  as  ” 252  (30.8),”  the  first  number  represents  the 
percentage  of  all  respondents,  and  the  second  number  in  parenthesis  represents  the 
percentage  of  just  those  respondents  who  responded  to  that  particular  question. 
This  annotation  will  be  found  on  those  questions  for  which  more  than  one 
response  was  allowed.) 
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Appendix  B-l 

Training  Manager 
Survey  Form 


87 


Squadron  Level  Training 
Survey  Form 
For  Training  Managers 


Prepared  For:  Technical  Training  Research  Division 

Human  Resources  Directorate  of  the 
Armstrong  Laboratory 
Brooks  A.F.B.,  TX  78235-5601 


Prepared  By:  HAY  Systems,  Inc. 

Suite  650 

2000  M  Street,  N.W. 
Washington,  DC  20036 


1  November  1991 


Version  Number: 

0002 
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To  All  Survey  Participants: 


You  have  been  selected  to  take  part  in  a  survey  covering  various  aspects  of  training 
in  the  Air  Force.  This  project  is  being  sponsored  by  the  Technical  Training  Research 
Division,  Human  Resources  Directorate  of  the  Armstrong  Laboratory,  Brooks  AFB, 
TX. 

The  information  will  be  used  for  research  on  training  at  Operational  Units  in  the 
Air  Force.  Rapidly  changing  technology  in  a  changing  world  requires  that  Air  Force 
training  be  responsive.  Your  assistance  with  this  survey  will  help  us  better  un¬ 
derstand  the  viewpoint  of  those  involved  in  base  level  training  and  the  on-site 
training  process. 

The  information  you  provide  will  be  kept  in  strictest  confidence.  The  background 
information  that  you  are  asked  to  provide  on  the  answer  sheet  will  be  used  to  help 
assure  that,  overall,  the  people  completing  this  survey  are  representative  of  the  total 
population  in  the  Air  Force. 


Privacy  Act  Statement 

Authority:  5  USC  Sec  301,  E09397,  and  AFR  35-2 

Principal  Purpose:  Collection  of  information  concerning  training  program 
development  and  evaluation.  SSAN  for  positive  identifications. 

Routine  Area:  Personnel  research  and  personnel  management  system  applications. 

Disclosure:  Failure  to  complete  this  requirement  will  detract  from  the  Air  Force’s 
capability  to  carry  out  programs  listed  above. 
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Squadron  Level 
Training  Survey  Form 


Instructions  For  Completing  The  Questionnaire 


•  You  should  have  two  sharpened  pencils  and  an  eraser. 

•  Mark  all  answers  on  the  answer  sheet  only. 


Do  Not  Mark  On  The  Survey  Forms 


Wrong 

gp  co  cio 

Wrong 

CD  CD  GD 

Wrong 

CZD  OTDCD 

Right 


•  Make  heavy  black  marks  that  fill  the  oval  completely. 
(See  examples  at  right,  which  show  both  correctly  and 
incorrectly  marked  answer  spaces.) 

•  Fill  in  just  one  oval  unless  you  are  given  instructions  to 
mark  all  that  apply. 

•  If  you  wish  to  change  an  answer,  erase  completely. 

•  Make  no  stray  marks  on  the  answer  sheet. 


Filling  In  The  Answer  Sheet 


Do  Not  fill  In  Your  Name  or  Date  of  Birth 

Before  answering  any  of  the  survey  questions,  fill  in  the  following  information  in  the  spaces  as  indicated. 
Name  Grid: 

•  Using  LIST  A  on  the  following  page,  enter  the  letter  that  matches  the  major  command,  field  operating 
agency,  or  direct  reporting  unit  in  the  first  column  of  the  Last  Name  section  of  your  answer  sheet 

•  Using  LIST  B  on  the  following  page,  enter  the  letter  that  matches  the  type  of  unit  to  which  you  are 
assigned  in  the  first  column  of  the  First  Name  section  of  your  answer  sheet. 

Social  Security  Number  Grid: 

•  Fill  in  your  Social  Security  Number. 

"Grid"  Grid: 

•  Fill  in  the  Version  Number  located  in  the  lower  right  comer  of  the  Survey  form  cover  sheet;  use  all  four 
columns,  for  example.  0004. 

Be  sure  to  fill  in  the  information  as  shown  in  the  Grid-Marking  Example  shown  on  the  lower  right  hand 
corner  of  side  one  of  your  answer  sheet. 

Starting  in  row  1  of  your  answer  sheet  fill  in  your  responses.  This  survey  should  take  you  less  than  one  hour 
to  complete. 

After  completion  of  this  survey,  return  the  survey  form  and  the  answer  sheet  to  the  person  who  gave  you 
these  materials 
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LIST  A 


AIR  FORCE  MAJOR  COMMANDS,  FIELD  OPERATING  AGENCIES,  AND  DIRECT 

REPORTING  UNITS 


Enter  the  LETTER  that  matches  your  selection 
section  of  your  answer  sheet. 


A.  Air  Force  Logistics  Command 
(AFLC)  25 

B.  Air  Force  Systems  Command 
(AFSC)  35 

G  Air  Force  Space  Command 
(AFSPACECOM)  03 

D.  Air  Training  Command  (ATC)  3.8 

K  Air  University  (AU)  4.8 

F.  Electronic  Security  Command  (ESC) 

0.8 

G.  Military  Airlift  Command  (MAC) 
16.8 

H.  Strategic  Air  Command  (SAC)  22.6 

I.  Tactical  Air  Command  (TAC)  15.6 

J.  Air  Force  Combat  Operations  Staff 
(AFCOS)  03 

K.  Air  Force  Communications  Agency 
(AFCA)  05 

L.  Defense  Agency  0.0 

M.  Air  Force  District  of  Washington 
(AFDW)  0.0 

N.  Air  Force  Inspection  Center  (AFIC) 
03 

O.  Air  Force  Intelligence  Agency 
(AHA)  0.0 


on  the  first  Column  of  the  Last  Name 


P.  Air  Force  Legal  Services  Center 
(AFLSC)  0.0 

Q.  Air  Force  Management  Engineering 
Agency  (AFMEA)  0.0 

R.  Air  Force  Military  Personnel  Center 
(AFMPC)  03 

S.  Air  Force  Morale,  Welfare,  and 
Recreation  Agency  (AFMWRA)  0.3 

T.  Air  Force  Office  of  Medical  Support 
(AFOMS)  0.0 

U.  Air  Force  Office  of  Security  Police 
(AFOSP)  03 

V.  Air  Force  Office  of  Special  Investiga¬ 
tions  (AFOSI)  0.0 

W .  Air  Force  Operational  Test  and 
Evaluation  Center  (AFOTEC)  0.0 

X.  Air  Force  Safety  Center  0.0 

Y.  US  Air  Force  Academy  (USAF A)  0.0 

Z.  Other  27.6 

(USAFE  17.8) 

(PACAF  2.8) 
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LIST  B 


DESCRIPTION  OF  UNIT  OF  ASSIGNMENT 

Enter  the  LETTER  that  matches  your  selection  on  the  first  Column  of  the  First 
Name  section  of  your  answer  sheet. 


A.  Aircraft  Maint  17.0 

B.  Audiovisual  0.5 
G  Audit  0.0 

D.  Chaplain  03 

E.  Civil  Engnrg  5-3 

F.  Communications  6.6 

G.  Comptroller  23 

H.  Education  2.8 

I.  Flying  Sqn  4.1 


J.  Info  Mgmt  15 

K.  Legal  0.0 

L  Manpower  0.0 

M.  Medical  3J 

N.  Missile  Sqn  15 

O.  MWR  0.0 

P.  Personnel  33 

Q.  Public  Affairs  0.0 

R.  Safety  0.0 


S.  Security  Police  4.3 

T.  Services  2.5 

U.  Social  Actions  03 

V.  Supply  4.1 

W.  Transportation  3.8 

X.  Training  193 

Y.  Weather  0.5 

Z.  Other  16.8 
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Survey  for  Training  Managers 

Section  I 


Section  I  -  Demographics 

Please  do  not  proceed  until  you  have  completed  the  section  entitled  Tilling  In  The  Answer  Sheet”. 


1.  Your  grade: 

a.  E-3  Airman  First  Class  0.0 

b.  E-4  Senior  Airman/Sergeant  13.8 
c  E-5  Staff  Sergeant  373 

d.  E-6  Technical  Sergeant  26.8 


e.  E-7  Master  Sergeant  18.5 

f.  E-8  Senior  Master  Sergeant  1.8 

g.  E-9  Chief  Master  Sergeant  0.5 

h.  G-S  Civilian  1.5 

i.  W-G  Civilian  0.0 


2.  Years  on  active  duty  (Military);  years  with  Civil  Service  (Civilians): 

a.  1  to  4  years  1.0 

b.  4  to  8  years  153 
c  8  to  12  years  28.9 

d.  12  to  16  years  29.1 

e.  Over  16  years  25.6 


3.  Your  current  role  in  the  training  process  is:  (Mark  all  that  apply) 

a.  Squadron  training  monitor  23.5 

b.  Section  training  monitor/coordinator  12.3 

c.  Squadron  OJT  monitor  22.8 

d.  Section  OJT  monitor/coordinator  9.3 

e.  Instructor  (Classroom,  does  not  include  OJT  Trainer)  15.0 

f.  Unit  Training  Manager  75.0 

g.  Base  Training  Manager  8.5 

h.  Ancillary  Training  Manager  30.5 

i.  Civilian  Training  Manager  6.3 

j.  Formal  Training  Monitor  22.8 

k.  I  have  no  training  role  at  this  time  2.S 


4.  Years  in  Training  Role: 

a.  Less  than  1  year  12.3 

b.  1  to  4  years  35.3 
c  4  to  8  years  24.3 

d.  8  to  12  years  17.0 

e.  over  12  years  11.0 
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Survey  for  Training  Managers 

Section  I 


5.  Is  your  role  in  the  training  process  your  primary  job  or  an  additional  duty? 

a.  Primary  job  71.9 

b.  Additional  duty  28.1 
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Survey  for  Training  Managers 

Section  II 


Section  II  -  Your  Functions  In  Managing  Training 


This  survey  asks  about  several  types  of  activities  in  managing  and  delivering 
training  at  the  unit  level.  Please  indicate  which  functions  are  a  part  of  your 
responsibilities.  Separate  sections  of  the  questionnaire  apply  to  each  function. 


6.  Which  of  the  following  tasks  do  you  do?  (Mark  all  that  apply! 

a.  Schedule  students  into  training.  78.3 

b.  Develop  training  courses  or  exercises.  34.5 
c  Conduct  formal  or  informal  training.  60.0 

d.  Assist  supervisors  with  OJT  training.  84.0 

e.  Document  training  completed  by  individuals.  63.8 

f.  Report  training  activity.  64.0 

g.  Evaluate  individual  performance.  41.5 

h.  Evaluate  training  system  performance.  55.0 

i.  Conduct  long  range  planning  for  training.  56.0 

j.  Assess  the  need  for  new  training  programs.  63.3 


7.  Approximately  what  portion  of  your  duty  time  is  normally  spent  on  training, 
including  both  conducting  training  and/or  management?  (Mark  one  response 
only) 

a.  All  or  nearly  all  48.4 

b.  About  three-quarters  16.5 

c.  About  half  11.3 

d.  About  one-quarter  9.8 

e.  About  five  to  ten  percent  7.0 

f.  Less  than  five  percent  7.0 
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Survey  for  Training  Managers 

Section  III 


Section  III  -  Scheduling 


Questions  8*12  apply  to  individuals  who  are  involved  in  scheduling  students  for 
training,  either  within  or  outside  the  duty  section.  If  you  Ho  not  routinely  schedule 
people  for  training,  please  skip  to  Section  IV. 

348  Respondents  of  400  =  87%  (First  percent  listed  is  of  the  total  400.  Percent  in 
parenthesis  is  of  the  348  who  actually  responded) 


8.  Which  scheduling  functions  do  you  normally  do?  (Mark  all  that  apply) 

a.  Schedule  personnel  for  training.  76.3  (87.6) 

b.  Schedule  class  times  and  locations.  53.8  (61.8) 

c  Schedule  the  use  of  equipment  (e.g.,  aircraft,  simulators,  computers,  etc.)  for 
instructional  use.  31.8  (36.5) 

d.  Notify  students  of  scheduled  ancillary  training.  57.3  (65.8) 

e.  Notify  students  of  scheduled  Enlisted  Specialty  Training  (EST).  63.0  (72.4) 

f.  Send  reminder  notices  to  students.  47.3  (54.3) 

g.  Plan  make-up  sessions  for  students  who  miss  a  training  session.  42.3  (48.6) 

h.  None  of  the  above.  5.0  (5.7) 


9.  About  what  portion  of  your  duty  time  is  normrlly  spent  on  scheduling? 

a.  All  or  nearly  all  3.4  d.  About  one-quarter  27.3 

b.  About  three-quarters  5.5  e.  About  5  to  10  percent  23.0 

c  About  half  16.1  f.  Less  than  5  percent  24.7 


10.  How  would  you  describe  the  systems  you  currently  use  to  do  your  scheduling? 
(Select  one) 

a.  Fully  automated.  7.8 

b.  Automated  scheduling  from  manual  input.  39.2 

c  Manual  scheduling  from  automated  input  (source  documents).  27.1 
d.  Fully  manual.  25.9 
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Survey  for  Training  Managers 

Section  III 


11.  What  are  the  manual  sources  of  information  you  use  in  scheduling?  (Mark  all 
that  apply) 

a.  AF  Form  2426  49.8  (57.2) 

b.  AF  Form  1320/ 1320a  (Training  Chart)  4.3  (4.9) 

c  AF  Form  403  (Request  for  Special  Technical  Training)  16.3  (18.7) 

d.  Letters /memos  of  request  64.0  (73.6) 

e.  Workcenter  rosters  31.0  (35.6) 

f.  Printouts  received  from  some  other  source  46.3  (53.2) 

g.  DD  Form  1556  13.5  (15.5) 

h.  Other  21.5  (24.7) 

i.  None  1.8  (2.0) 


12.  How  adequate  are  the  sources  of  information  that  you  currently  use  for 
scheduling?  (Mark  one  response  only) 

a.  Completely  adequate  45.0 

b.  Somewhat  adequate  48.7 

c.  Somewhat  inadequate  5.4 

d.  Completely  inadequate  0.9 
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Survey  for  Training  Managers 

Section  III 


Please  use  the  following  scale  to  rate  each  of  the  aspects  of  scheduling  in  items  13  to 
19. 


a.  Not  a  problem  at  present  -  no  improvement  needed. 

b.  Only  minor  problems  with  this  at  present. 

c  Some  problems  with  this  -  improvement  desirable. 

d.  Major  problems  with  this  -  improvement  urgently  needed. 

e.  Not  relevant  to  scheduling. 


13.  Access  to  existing  data  systems  needed  for  scheduling. 

a.  393  d.  6.0 

b.  223  e.  11.1 

c  21.4 


14.  The  design  and/or  operation  of  an  automated  system  used  for  scheduling. 

a.  33.3  d.  9.7 

b.  20.2  e.  163 

c  20.2 

15.  Extent  of  standardization  in  course  titles/codes. 

a.  42.0  d.  8.0 

b.  15.1  e.  18.0 

c  16.9 

16.  Integrating  scheduling  with  other  training  functions,  such  as  record  keeping  and 
reporting. 

a.  35.0  b.  8.6 

b.  23.2  e.  12.0 

c.  21.2 

17.  The  extent  of  your  training/preparation  for  doing  scheduling  functions. 

a.  47.4  d.  5.7 

b.  29.0  e.  3.4 

c  14.4 


102 


Survey  for  Training  Managers 

Section  III 


18.  Getting  , people  to  attend  training  as  scheduled. 

a.  25.4  d.  10.9 

b.  38.6  e.  1.4 

c  23.7 

19.  Coordinating  training  schedules  with  other  unit  activities. 


a. 

35.8 

d. 

4.6 

b. 

35.0 

e. 

3.4 

c 

21.2 
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Survey  for  Training  Managers 

Section  IV 


Section  IV  -  Training  Development 

This  section  applies  to  individuals  who  currently  plan  or  develop  any  training 
courses  or  exercises,  either  ancillary  or  job-related.  Training  development  includes 
writing  training  objectives,  testing  standards,  or  tests;  developing  instructional 
materials;  and  other  tasks  directly  related  to  creating  training  materials.  At  the  unit 
level,  this  may  be  in  special  job-related  training  programs  such  as  FTD  or  AMQP, 
ancillary  training  such  as  Self/ Buddy  Care  or  CPR,  or  special  training  directed  by 
Commander/Supervisor  such  as  introducing  new  equipment  or  policy. 

If  you  are  not  involved  in  the  development  of  training,  go  to  Section  V. 

(225  (56%)  responded) 

20.  About  what  portion  of  your  duty  time  is  normally  spent  on  training  develop¬ 
ment? 

a.  All  or  nearly  all  6.2  d.  About  one-quarter  23.1 

b.  About  three-quarters  4.9  e.  About  5  to  10  percent  29.8 

c  About  half  12.0  f.  Less  than  5  percent  24.0 


21.  Have  you  developed  or  assisted  in  developing  any  of  the  following?  (Mark  all 

thatapply) 

a.  Formal  course(s)  for  Enlisted  Specialty  Training  (EST)  10.0  (17.8) 

b.  Formal  courses  for  ancillary  training  12.5  (22.2) 
c  Methods  for  on  job  training  (OJT)  25.8  (63.6) 

d.  Exercises  or  examples  to  supplement  formal  courses  18.3  (32.4) 

e.  Mock-up  or  example  equipment  for  use  in  training  11.3  (20.0) 

f.  Other  22.5  (40.0) 
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22.  What  sources  do  you  use  to  determine  training  content?  (Mark  all  that  apply) 

a.  STS/AF  Form  797  44.0  (78.2) 

b.  AFJQS  243  (43.1) 

c  Command  JQS  19.3  (34.2) 

d.  Master  Task  List  36.3  (64.4) 

e.  Occupational  Survey  Report  5.3  (9.3) 

f.  Unit  level  task  analysis  18.8  (333) 

g.  MAJCOM  training  plans  12.5  (22.2) 

h.  Talking  to  functional  managers  about  need  for  training  21.0  (37.3; 

i.  Past  training  evaluation  reports  193  (34.2) 

j.  AF  Form  1098  13.8  (24.4) 

k.  Technical  orders  CTOs)  18.5  (32.9) 

23.  About  what  portion  of  the  training  materials  used  in  your  unit  are  developed 
within  your  unit? 

a.  All  or  nearly  all  14.4  d.  About  one-quarter  19.0 

b.  About  three-quarters  15.7  e.  About  5  to  10  percent  26.4 

c  About  half  20.8  f.  None  3.7 


24.  Which  of  the  following  steps  do  you  usually  do  when  developing  training? 

(Mark  all  that.applyl 

a.  Conduct  a  training  needs  assessment  38.0  (67.6) 

b.  Select  tasks  requiring  training  383  (68.0) 
c  Develop  objectives  373  (66.2) 

d.  Develop  tests  28.8  (51.1) 

e.  Plan  sequence  of  instruction  32.5  (57.8) 

f.  Select  instructional  methods  34.0  (60.4) 

g.  Select  instructional  media  28.8  (51.1) 

h.  Determine  resource  and  funding  requirements  22.0  (39.1) 

i.  Develop  instructional  materials  33.8  (60.0) 
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Section  V  -  Documentation 


Questions  25  to  28  deal  with  maintaining  documentation  of  student  training 
activities  (record  keeping).  If  you  do  not  routinely  document  training  activity,  skip 
to  Section  VI.  (320  (80%)  responded) 

25.  About  what  portion  of  your  time  is  normally  spent  on  documentation? 

a.  All  or  nearly  all  3.4  d.  About  one-quarter  28.1 

b.  About  three-quarters  5.3  e.  About  5  to  10  percent  31.6 

c  About  half  11.9  f.  Less  than  5  percent  19.7 


26.  How  would  you  describe  the  system  you  currently  use  to  document  completed 
training? 

a.  Fully  automated.  20.7 

b.  Partially  automated.  55.5 

c.  Fully  manual.  23.8 


27.  Is  the  written  guidance  for  your  documentation  activities  clear  and  easy  to 
understand? 

a.  Quite  clear  42.0 

b.  Somewhat  clear  32.9 

c.  Somewhat  unclear  10.3 

d.  Very  unclear  7.5 

e.  None  available  7.2 


28.  What  hard-copy  sources  of  information  do  you  use  for  documentation?  (Mark 

all  that  apply) 

a.  AF  Form  2426  47.0  (58.8) 

b.  Student/ course  training  roster  42.8  (53.4) 
c  Letters /memos  of  completion  46.8  (58.4) 

d.  Printouts  received  from  some  other  source  46.5  (58.1) 

e.  Other  29.0  (36.3) 
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29.  What  automated  information  systems  do  you  use  in  documentation?  (MflrK  all 
that  apply) 

a.  Personnel  Data  System  (PDS)  13.0  (163) 

b.  CAMS  33.5  (41.9) 
c  MMICS  1.8  (2.2) 

d.  CASB  0.3  (0.3) 

e.  MAJCOM-spedfic  information  system  6.3  (7.8) 

f.  Locally  devised  data  base  24.8  (30.9) 

g.  Other  14.0  (17.5) 

h.  None.  I  do  not  use  automated  information  systems  14.0  (17.5) 


Please  use  the  following  scale  to  rate  each  aspect  of  your  documentation  activity 
listed  in  Items  30  to  34.  (Choose  one  response  for  each  item) 

a.  Not  a  problem  at  present  -  no  improvement  needed. 

b.  Only  minor  problems  with  this  at  present. 

c  Some  problems  with  this  -  improvement  desirable. 

d.  Major  problems  with  this  -  improvement  urgently  needed. 

e.  Not  relevant  to  documentation. 


30.  Access  to  existing  data  systems. 

a.  42.5  d.  6.8 

b.  26.4  e.  12.1 

c  12.1 


31.  Design  and/or  operation  of  automated  system  used  for  documentation. 

a.  35.8  d.  10.6 

b.  253  e.  14.3 

c  14.0 

32.  Extent  of  standardization  in  course  titles/codes. 


a. 

40.8 

d. 

8.1 

b. 

18.1 

e. 

193 

c 

13.7 
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33.  Integrating  documentation  with  other  training  functions,  such  as  scheduling 
and  reporting. 

a.  37.6  d.  9.6 

b.  26.1  e.  11.2 

c  15.5 

34.  Obtaining  the  needed  information  on  attendance  at  training  activities. 

a.  37.9  d.  82 

b.  29.8  e.  82 

c.  16.0 
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Section  VI  -  Reporting 


Questions  35-39  apply  to  individuals  who  report  training  activity  either  by  stan¬ 
dardized  or  special  reports.  If  you  do  not  routinely  report  training  activity,  skip  to 
Section  VIL  <302  (76%)  responded) 

35.  About  what  portion  of  your  duty  time  is  normally  spent  preparing  and 
delivering  reports? 

a.  All  or  nearly  all  2.3  d.  About  one-quarter  19.9 

h.  About  three-quarters  1.0  e.  About  5  to  10  percent  38.1 

c.  About  half  5.0  f.  Less  than  5  percent  33.8 


36.  For  whom  do  you  prepare  reports:  (Mark  all  that  apply) 

a.  Squadron  level  62.3  (82.5) 

b.  Wing/base  level  44.0  (583) 
c  Numbered  Air  Force  5.0  (6.3) 

d.  MAJCOM  22.0  (27.5) 

e.  HQUSAF  2.5  (3.3) 

f.  Technical  School  2.8  (3.6) 

g  Other  outside  agencies  10.3  (12.8) 

37.  Do  you  feel  that  the  amount  of  reporting  done  in  regard  to  training  is:  (Select 
one) 

a.  Not  enough  8.6 

b.  About  right  67.4 
c  Too  much  23.9 


38.  Which  of  the  following  types  of  reports  do  you  deliver?  (Mark  all  that  apply) 

a.  Oral  (briefings)  47.8  (63.2) 

b.  Automated  (computer)  34.8  (46.0) 
c  Written  55.5  (73.5) 

d.  Standard  format  (fill  in  the  blanks)  30.5  (40.4) 
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39.  How  are  the  reports  that  you  prepare  used?  (Mark  all  that  apply) 

a.  Remind  airmen  of  their  ancillary  training  requirements/schedules.  30.5 
(40.4) 

b.  Check  EST  received  against  the  required  standards.  30.8  (40.7) 
c  Analyze  die  effectiveness  of  training  delivery 35.8  (47.4) 

d.  Analyze  the  trends  in  delivery  of  training.  30.8  (40.7) 

e.  Determine  additional  training  requirements.  38.8  (51.3) 

f.  Determine  training  equipment  requirements.  13.3  (17.5) 

g.  None  of  the  above,  or  don't  know.  13.8  (18.2) 
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Section  VII  -  Evaluation 


This  section  deals  with  the  evaluation  of  individual  training  or  training  .courses.  It 
does  not  refer  to  the  initial  evaluation  of  individuals  entering  Upgrade  Training 
(UGT).  It  does  refer  to  the  evaluation  of  tasks  leading  to  certification  in_UGT  plus 
any  other  formal  evaluation  of  training.  (252  (63%)  responded) 

If  you  do  not  currently  evaluate  training,  skip  to  Section  VUI. 

40.  About  what  portion  of  your  duty  time  is  spent  on  testing/evaluation  activities? 

a.  All  or  nearly  all  2.4  d.  About  one-quarter  15.9 

b.  About  three-quarters  2.8  e.  About  5  to  10  percent  38.1 

c  About  half  9.5  f.  Less  than  5  percent  31.3 


41.  Which  of  the  following  evaluation  tasks  do  you  do?  (Mark  all  that  apply) 

a.  Administer/monitor  performance  tests  28.3  (44.8) 

b.  Administer /monitor  written  tests  38.8  (61.5) 

c.  Score  tests  34.0  (54.0) 

d.  Record  test  results  36.3  (575) 

e.  Report  test  results  31.0  (49.2) 

f.  Develop/write  test  items  17.5  (27.8) 

g.  Design/assemble  tests  16.5  (26.2) 


42.  What  sources  of  information  are  used  to  evaluate  the  delivery  of  training? 

(Maik.all  that  apply) 

a.  Student  critiques  31.5  (50.0) 

b.  Course  control  documents  (STS,  JQS,  POI)  27.0  (42.9) 
c  Test  scores  of  students  41.5  (65.9) 

d.  Job/task  evaluations  of  students  33.0  (52.4) 

e.  Inspection  results  (IG,  ORI,  QA,  etc.)  21.5  (34.1) 

f.  ISD  reports /records  4.8  (7.5) 

g.  Local  checklist/procedure  26.8  (42.5) 

h.  Other  115  (18.3) 
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43.  What  sources  of  information  are  used  to  evaluate  the  results  of  training?  (Mark 
all  that  apply) 

a.  Student  critiques  293  (46.4) 

b.  Course  control  documents  (STS,  JQS,  POD  253  (40.1) 

c.  Test  scores  of  students  413  (65.51 

d.  Job/task  evaluations  of  students  35.0  (55.6) 

e.  Inspection  results  (IG,  ORI,  QA,  etc.)  23.0  (36.5) 

f.  ISD  reports/records  4.0  (6.3) 

g.  Local  checklist/procedure  24.8  (393) 

h.  Other  8.8  (13.9) 


44.  What  measures  are  used  to  assess  performance  for  the  training  you  administer? 
(Mark  all  that  apply) 

a.  Observe  graduate  performance.  26.0  (41.2) 

b.  Question  graduates  about  job  performance.  203  (32.1) 
c  Formal  performance.  17.5  (27.8) 

d.  Interview  supervisors  of  graduates.  21.0  (33.3) 

e.  Written  exams.  21.8  (34.5) 

f.  No  performance  measures  are  used.  10.3  (163) 

g.  Other.  6.3  (9.9) 


45.  What  information,  assistance,  or  methods  would  you  like  to  have  for  use  in 
your  evaluation  duties?  (Mark  all  that  apply) 

a.  Formal  methods  of  determining  student  performance.  13.3  (21  0) 

b.  Further  training  for  how  to  evaluate  performance.  19.5  (31.0) 

c  A  job-aid,  such  as  computer  program  or  video,  to  explain  training 
evaluation.  273  (433) 

d.  A  comprehensive  checklist  of  what  to  evaluate.  25.3  (40.1) 

e.  Evaluation  built  into  the  course  and  certified  by  the  instructor.  16.0  (25.4) 

f.  More  explicit  performance  standards  for  the  AFSCs  in  my  unit.  20.8  (32.9) 

46.  What  computer  based  information  systems  or  tools  do  you  use  in  managing 
data  from  training  evaluations?  (Mark  all  that  apply) 

a.  None  24.3  (383) 

b.  Standard  data  base  managing  tools,  such  as  Lotus  1-2-3,  dBase  or  SAS  10.0 
(15.9) 

c  CAMS  15.8  (25.0) 

d.  MMICS  1.0  (1.6) 

e.  Locally  devised  system  153  (24.2) 

f.  Other  9.8  (15.5) 
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Section  VHt  -  Opinions  About  Training 


This  section  asks  for  your  own  opinions  about  some  aspects  of  the  training  that 
occurs  within  vour  unit.  Please  use  the  following  scale,  and  choose  one  response  for 
each  item. 

a.  Strongly  Agree 

b.  Agree 

c  Disagree 

d  Strongly  Disagree 

e.  Not  applicable  to  your  unit;  or  Do  not  know. 

47.  Standards  for  training  in  my  unit  go  far  beyond  general  Air  Force  standards. 

a.  6.6  d  10.0 

b.  33.9  e.  63 

c  433 

48.  When  technologies  for  managing  training  (e.g.,  automated  systems)  are 
introduced  in  our  unit,  they  are  used  to  full  capacity. 

a.  14.7  d  7.1 

b.  38.4  e.  17.4 

c  22.4 

49.  For  the  jobs  in  my  unit,  hands-on  training  is  generally  more  effective  than 
classroom  training. 

a.  39.4  d.  3.1 

b.  39.6  e.  3.7 

c  14.2 

50.  Computer-assisted  training  technologies  can  be  as  effective  as  classroom 
training. 

a.  19.4  d.  63 

b.  40.8  e.  113 

c.  22.0 

51.  My  unit  has  sufficient  resources  to  provide  more  training  than  it  is  currently 
conducting 


a.  9.4 

b.  25.9 

c.  42.9 


d  13.1 
e.  8.6 
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52.  For  most  AFSCs  in  my  squadron,  developing  formal  training  courses  could 
frequently  be  done  at  the  squadron  level. 

a.  65  d.  17.2 

b.  26.6  e.  11.2 

c  38.5 

53.  Training  packages  developed  outside  the  unit  (for  example,  at  headquarters 
level  or  by  contractors)  are  generally  useful  for  squadron  level  training 

a.  7.0  d.  5.0 

b.  48.8  e.  23.2 

c  15.9 

54.  Contractors'  technical  representatives  fill  a  crucial  role  in  the  accomplishment 
of  technical  training  requirements  within  my  squadron. 

a.  6.3  d.  11.1 

b.  17.9  e.  42.0 

c  22.7 

55.  Training  devices  and  simulators  are  frequently  used  for  training  in  our  unit. 

a.  9.9  d.  9.9 

b.  25.3  e.  28.6 

c  26.3 

56.  Training  devices  and  simulators  are  as  effective  as  hands-on  training  using 
actual  equipment 

a.  14.3  d.  63 

b.  35.2  e.  21.6 

c  22.7 
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Section  IX  -  Future  Training 


The  following  section  describe,  a  possible  future  situation. 

At  some  time  in  the  future  the  Air  Force  directs  that,  for  most  AFSCs,  time  available 
for  formal  technical  school  training  will  be  reduced  by  half.  Substantial  training 
responsibilities  are  to  be  shifted  to  the  base  and  squadron  level.  In  order  to  ac¬ 
complish  this  new  tasking,  base  and  squadron  training  and  training  management 
capabilities  need  to  be  enhanced. 

Answer  the  following  questions  assuming  that  YOU  have  been  tasked  to  help  deter¬ 
mine  requirements  at  your  base  for  supporting  increased  training  responsibilities. 

As  usual,  these  increased  responsibilities  must  be  supported  with  as  small  an 
increase  in  training  manpower  as  possible.  Do  not  base  your  responses  on  current 
regulations,  operating  policies,  or  capabilities. 


57.  As  part  of  the  changeover  to  greater  base  and  squadron  level  training  respon¬ 
sibilities,  primary  responsibility  for  several  tasks  involved  in  determining 
training  requirements  will  be  moved  to  the  base  or  squadron  level.  Which 
single  task  do  you  feel  would  most  need  highly  modem  computer  based  data 
management  tools  or  techniques  in  order  to  be  carried  out  effectively  at  the  base 
or  squadron  level?  (Select  one) 

a.  Conduct  a  training  needs  assessment  21.6 

b.  Analyze  job  performance  requirements  23.2 

c.  Select  tasks  requiring  training  8.8 

d.  Determine  student  prerequisites  2.9 

e.  Select  appropriate  training  settings  1.9 

f.  Forecast  resource/logistics  requirements  15.5 

g.  Do  not  know  21.6 
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58-60  As  part  of  the  restructuring  of  training,  the  Air  Force  has  directed  base  and 
squadron  level  training  organizations  to  develop  in-house  capabilities  to  perform  a 
number  of  tasks.  Please  use  the  following  list  to  answer  items  57-59. 

a.  Develop  training  objectives 

b.  Develop  knowledge  tests 
c  Develop  performance  tests 

d.  Plan  sequence  of  instruction 

e.  Select  instructional  methods 

f.  Select  instructional  medias 

g.  Develop  instructional  materials 

h.  Validate  the  approach  and  content  used 

i.  Do  not  know. 

58.  If  your  unit  were  directed  to  perform  all  the  above  tasks,  which  of  the  tasks 
would  be  completely  new  responsibilities  to  the  unit?  (Mark  all  that  apply) 


a. 

21.8 

f. 

24.8 

b. 

315 

& 

36.0 

c 

273 

h. 

49.8 

d. 

23.0 

i. 

165 

e 

193 

59.  Which  one  of  the  tasks  listed  above  would  be  the  single  most  difficult  to 

implement  without  additional  manpower,  training,  or  equipment?  (Select  one) 


a.  5.3 

b.  6.7 
c  2.5 
d.  2.5 
e  2.0 


f.  1.7 

g.  39.0 

h.  16.0 

i.  24.2 


60.  For  each  of  the  tasks  listed  above,  mark  the  ones  which  would  require  training 
personnel  to  receive  additional  training  to  perform.  (Mark  all  that  apply) 


a. 

34.8 

f.  30.3 

b. 

433 

g.  53.0 

c 

39.8 

h.  51.8 

d. 

34.0 

i.  14.8 

e 

29.8 
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61-62  As  part  of  the  new  training  strategy,  the  Air  Force  has  directed  that  OJT  be  the 
primary  method  of  job  skills  training.  You  have  been  tasked  to  provide  additional 
training  to  all  OJT  instructors  on  teaching  and  assessment  methods  in  the  work 
environment. 


61.  If  you  were  free  to  select  any  means  of  instructional  delivery  to  teach  OJT  instruc¬ 
tors  (train  the  trainers),  which  of  the  following  would  you  believe  to  be  most 
efficient  in  terms  of  scheduling,  record  keeping,  and  other  administrative  tasks? 

(Select  .one) 

a.  instructor-led  classroom  instruction  46.7 

b.  Videotaped  "lectures"  7.9 

c  Computer-based  training  25.7 

d.  Self  study  printed  materials  3.9 

e.  One-on-one  work  with  OJT  instructors  15.7 


62.  Which  one  of  the  following  means  of  instructional  delivery  would  you  feel  to 
be  the  most  effective  for  preparing  OJT  instructors  to  carry  out  their  duties? 
(Select,  one) 

a.  Instructor-led  classroom  instruction  52.0 

b.  Videotaped  "lectures"  4.7 

c  Computer-based  training  12.1 

d.  Self  study  printed  materials  2.1 

e.  One-on-one  assistance  to  OJT  instructors  29.1 


63-66  In  support  of  the  increased  requirements  for  base  and  squadron  level  training, 
you  are  tasked  to  prepare  a  plan  to  support  training.  Please  use  the  following  list  to 
answer  items  63-66. 

a.  Scheduling  students 

b.  Scheduling  resources 

c  Tracking  student  progress 

d.  Managing  training  requirements 

e.  Maintaining  instructional  materials  and  equipment 

f.  Updating  instructional  materials  and  equipment 

g.  None  of  the  above 
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63.  For  which  of  the  tasks  would  you  propose  to  continue  to  use  current  methods 
and  procedures  with  little  or  no  modification?  (Mark  all  that  apply) 


a.  533  e.  29.8 

h  363  f.  193 

c.  36.8  g  183 

d.  353 


64.  For  each  of  the  tasks  listed  above,  mark  the  ones  which  would  require  training 
personnel  to  receive  additional  training  to  perform.  (Mark  all  that  apply) 

a.  9.8  e.  36.0 

b.  20.8  f.  58.0 

c.  24.0  g.  21.0 

d.  373 

65.  For  which  single  task  from  the  above  list  would  automated  record  keeping 
prove  to  be  most  beneficial,  given  increased  base  and  unit  training 
requirements?  (Select  one) 

a.  26.6 

b.  43 

c  29.8 
d.  24.9 

66.  For  which  single  task  from  the  above  list  would  you  suggest  that  additional 

manpower  would  be  needed  regardless  of  the  level  of  automation  provided? 

(Select  one) 

a.  5.0 

b.  2.8 
c  6.1 
d.  193 

67-69  In  support  of  the  increased  requirements  for  base  and  squadron  level  training, 
evaluation  responsibilities  will  be  substantially  increased.  Please  use  the  following 
list  to  answer  items  66-68. 

a.  Evaluating  student  performance  during  training 

b.  Evaluating  student  performance  on  completion  of  training 
c  Evaluating  job  performance  during  training 

d.  Evaluating  job  performance  in  relation  to  training  performance 

e.  Measuring  the  usefulness  and  cost-benefits  of  various  local  training 
programs 

f.  Do  not  know 


e.  133 

f.  36.2 
g-  17.1 


e.  2.6 

f.  7.7 
g-  4.0 
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67.  Which  of  the  evaluation  responsibilities  do  you  believe  would  prove  to  be  the 
most  difficult  to  implement  at  the  base  or  squadron  level?  (Select  one) 


a. 

2.9 

d. 

14.9 

h 

1.6 

e. 

59.7 

c 

5.9 

f. 

14.9 

68.  Which  item  listed  above  would  require  the  fewest  changes  in  procedures 
currently  used  at  the  base  or  squadron  level?  (Select  one) 


29.0 

d. 

82 

26.1 

e. 

45 

16.0 

f. 

16  2 

69.  For  which  single  task  from  the  above  list  would  automated  record  keeping 
prove  to  be  most  beneficial,  given  increased  base  and  unit  training 
requirements?  (Select  one) 


a. 

6.7 

d. 

16.2 

b. 

14.6 

e. 

29.1 

c 

3.8 

f. 

29.6 
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Section  X  -  Your  Training  Activities  During 
Desert  Shield/Desert  Storm 


We  need  to  know  more  about  how  to  plan  for  unit-level  training  in  a  wartime 
environment.  Please  answer  this  section  in  terms  of  your  unit's  training  during  the 
period  of  the  conflict  with  Iraq,  from  August,  1990  to  February,  1991.  Answer  this 
section  whether  or  not  vou  were  deployed  to  the  Middle  East. 

70.  During  the  conflict  with  Iraq,  were  you:  (Mark  all  that  apply) 

a.  Deployed  to  the  Middle  East.  6.5 

b.  Deployed  from  CONUS  to  overseas,  but  to  a  location  other  than  Middle  East. 
2.5 

c  Deployed  from  overseas  to  CONUS.  0.5 

d.  Deployed  within  the  U.S.  1.0 

e.  Did  not  move  -  remained  in  current  location.  86.3 


71.  If  you  deployed  during  the  conflict,  was  this  deployment  with  your  unit  or 
individually? 

a.  Does  not  apply  -  did  not  deploy.  89.7 

b.  Deployed  individually,  not  with  my  unit.  2.4 
c  Deployed  with  part  of  my  unit.  6.8 

d.  Deployed  with  entire  unit.  1.1 


72.  During  the  period  of  the  conflict  (from  August  1990  to  February  1991)  did  you 
manage  any  of  the  following  types  of  training  activities?  (Mark  all  that  apply) 

a.  Normal  OJT  training.  66.0 

b.  Special  OJT  to  learn  how  to  operate/ maintain  equipment  in  the  desert 
environment.  6.0 

c  Special  unit  training  to  learn  how  to  live  in  the  desert  environment.  3.8 

d.  Additional  ancillary  training  for  threats  such  as  nuclear /chemical /biological 
warfare.  25.0 

e.  Training  for  working  and  living  within  an  Arabic  culture.  3.3 

f.  Normal  ancillary  training.  43.5 

g.  Did  not  manage  training  during  that  period.  22.5 
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73.  During  the  period  of  conflict,  what  part  of  your  duty  time  was  devoted  to 
training  management  duties? 


a.  All  or  nearly  all  37.3 

b.  About  three-quarters  10.4 
c  About  half  9.4 


d.  About  one-quarter  8.6 

e.  About  5  to  10  percent  12.8 

f.  None  21.4 


74-80  During  Desert  Shield/Storm,  what  changes  occurred  in  your  unit's  training? 
Please  use  the  following  responses  to  indicate  what  types  of  change  occurred  in  each 
aspect  of  training  listed  in  items  73  to  79.  (Mark  all  that  apply) 

a.  Decreased 

b.  Increased 

c  Waived  part  or  all  of  normal  activities 

d.  Postponed  some  or  all 

e.  Changed  method  for  doing  this 

f.  Do  not  normally  perform  this 

g.  No  changes  in  this  item 

74.  Formal  job  related  training. 


a.  28.5 

b.  11.5 
c  8.3 

d.  16.8 


e.  23 

f.  3.8 
R  393 


75.  Informal  job  related  training. 


a.  17.0 

b.  21.8 
c  6.5 

d.  9.8 


e.  3.8 

f.  3.0 

g.  41.0 


76.  Ancillary  training. 


a.  10.8 

b.  29.5 

c.  83 

d.  12.5 


e.  4.8 

f.  3.0 

g.  35.8 
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77.  Documentation  of  training. 

a.  153  e.  2.0 

b.  11.8  f.  23 

c  5.0  g.  543 

d.  93 

78.  Use  of  automated  training  management  systems. 

a.  123  e.  13 

b.  73  f.  153 

c  33  g.  533 

d.  5.0 

79.  Direct  combat-related  training. 

a.  6.0  e.  20 

b.  293  f.  243 

c  2.8  g.  30.0 

d.  23 


80.  Evaluation  of  training. 

a.  13.8  e.  33 

b.  10.5  f.  4.0 

c  53  g.  53.0 

d.  7.8 

Thank  you  for  completing  this  survey.  Please  return  this  survey  form  and  your 
answer  sheet  to  the  person  who  gave  them  to  you. 
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Squadron  Level  Training 
Survey  Form 
For  Unit-Level  Trainers 


Prepared  For:  Technical  Training  Research  Division 

Human  Resources  Directorate  of  the 
Armstrong  Laboratory 
Brooks  A.F.B.,  TX  78235-5601 


Prepared  By:  HAY  Systems,  Inc. 

Suite  650 

2000  M  Street,  N.W. 
Washington,  DC  20036 


Version  Number: 

1  November  1991  0003 
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To  All  Survey  Participants: 

You  have  been  selected  to  take  part  in  a  survey  covering  various  aspects  of  training 
in  the  Air  Force.  This  project  is  being  sponsored  by  the  Technical  Training  Research 
Division  Human  Resources  Directorate  of  the  Armstrong  Laboratory,  Brooks  AFB, 
TX. 

The  information  will  be  used  for  research  on  training  at  Operational  Units  in  the 
Air  Force.  Rapidly  changing  technology  in  a  changing  world  requires  that  Air  Force 
training  be  responsive.  Your  assistance  with  this  survey  will  help  us  better  un¬ 
derstand  the  viewpoint  of  those  involved  in  base  level  training  and  the  on-site 
training  process. 

The  information  you  provide  will  be  kept  in  strictest  confidence.  The  background 
information  that  you  are  asked  to  provide  on  the  answer  sheet  will  be  used  to  help 
assure  that,  overall,  the  people  completing  this  survey  are  representative  of  the  total 
population  in  the  Air  Force. 


Privacy  Act  Statement 

Authority:  5  USC  Sec  301,  E09397,  and  AFR  35-2 

Principal  Purpose:  Collection  of  information  concerning  training  program 
development  and  evaluation.  SSAN  for  positive  identifications. 

Routine  Area:  Personnel  research  and  personnel  management  system  applications. 

Disclosure:  Failure  to  complete  this  requirement  will  detract  from  the  Air  Force's 
capability  to  carry  out  programs  listed  above. 
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•  You  should  have  two  sharpened  pencils  and  an  eraser. 

•  Mark  all  answers  on  the  answer  sheet  only. 


Do  Not  Mark  On  The  Survey  Forms 

*  Make  heavy  black  marks  that  fill  the  oval  completely. 

(See  examples  at  right,  which  show  both  correctly  and 

Wrong 

go 

C_B_J 

incorrectly  marked  answer  spaces.) 

Wrong 

•  Fill  in  just  one  oval  unless  you  are  given  instructions  to 
mark  all  that  apply. 

(  A  ) 

L  b_J 

Wrong 

GO 

CZD 

(3D 

t  c  ) 

•  If  you  wish  to  change  an  answer,  erase  completely. 

•  Make  no  stray  marks  on  the  answer  sheet. 

CZD 

Right 

(  cj 

Filling  In  The  Answer  Sheet 

Do  Not  Fill  In  Your  Name  or  Date  of  Birth 

Before  answering  any  of  the  survey  questions,  fill  in  the  following  information  in  the  spaces  as  indicated. 
Name  Grid: 

•  Using  LIST  A  on  the  following  page,  enter  the  letter  that  matches  the  major  command,  field  operating 
agency,  or  direct  reporting  unit  in  the  first  column  of  the  Last  Name  section  of  your  answer  sheet 

•  Using  LIST  B  on  the  following  page,  enter  the  letter  that  matches  the  type  of  unit  to  which  you  are 
assigned  in  the  first  column  of  the  First  Name  section  of  your  answer  sheet 

Social  Security  Number  Grid: 

•  Fill  in  your  Social  Security  Number. 

"Grid"  Grid: 

•  full  in  the  Version  Number  located  in  the  lower  right  comer  of  the  Survey  form  cover  sheet;  use  all  four 
columns,  for  example,  0004. 

Be  sure  to  fill  in  the  information  as  shown  in  the  Grid-Marking  Example  shown  on  the  lower  right  hand 
comer  of  side  one  of  your  answer  sheet 

Starting  in  row  1  of  your  answer  sheet  fill  in  your  responses.  This  survey  should  take  you  less  than  one  hour 
to  complete. 

After  completion  of  this  survey,  return  the  survey  form  and  the  answer  sheet  to  the  person  who  gave  you 
these  materials. 
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Sr 


USTA 


AIR  FORCE  MAJOR  COMMANDS,  HELD  OPERATING  AGENCIES,  AND  DIRECT 

REPORTING  UNITS 


Enter  the  LETTER  that  matches  your  selection  on  the  first  Column  of  the  Last  Name 
section  of  your  answer  sheet. 


A.  Air  Force  Logistics  Command 
(AFLC)  3.6 

B.  Air  Force  Systems  Command 
(AFSC)  4.9 

C  Air  Force  Space  Command 
(AFSPACECOM)  0.0 
D.  Air  Training  Command  (ATC)  1.5 
E  Air  University  (AU)  2.5 

F.  Electronic  Security  Command  (ESC) 

3.6 

G.  Military  Airlift  Command  (MAC) 

12.6 

H.  Strategic  Air  Command  (SAC)  19.8 

I.  Tactical  Air  Command  (TAC)  16.5 

J.  Air  Force  Combat  Operations  Staff 
(AFCOS)  0.0 

K.  Air  Force  Communications  Agency 
(AFCA)  1.7 
Defense  Agency  0.0 
Air  Force  District  of  Washington 
(AFDW)  0.2 

N.  Air  Force  Inspection  Center  (AFIC) 

0.3 

O.  Air  Force  Intelligence  Agency 
(APIA)  0.3 


P.  Air  Force  Legal  Services  Center 
(AFLSO  0.2 

Q.  Air  Force  Management  Engineering 
Agency  (AFMEA)  0.0 

R.  Air  Force  Military  Personnel  Center 
(AFMPC)  0.0 

S.  Air  Force  Morale,  Welfare,  and 
Recreation  Agency  (AFMWRA)  0.1 

T.  Air  Force  Office  of  Medical  Support 
(AFOMS)  0.4 

U.  Air  Force  Office  of  Security  Police 
(AFOSP)  0.9 

V.  Air  Force  Office  of  Special  Investiga¬ 
tions  (AFOSI)  0.1 

W .  Air  Force  Operational  Test  and 
Evaluation  Center  (AFOTEC)  0.1 

X.  Air  Force  Safety  Center  0.0 

Y.  US  Air  Force  Academy  (USAFA)  0.0 

Z.  Other  29.7 

(USAFE  19.7%) 

(PACAF  7.0%) 
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DESCRIPTION  OF  UNIT  OF  ASSIGNMENT 

Enter  the  LETTER  that  matches  your  selection  on  the  first  Column  of  the  First 
Name  section  of  your  answer  sheet. 


A.  Aircraft  Maint  27.3 

B.  Audiovisual  0.1 
C  Audit  0.0 

D.  Chaplain  0.1 
E  Civil  Engnrg  7.0 

F.  Communications  9.3 

G.  Comptroller  1.1 

H.  Eduation  0.7 

I.  Flying  Sqn  2.7 


J.  Info  Mgmt  2.5 

K.  Legal  0.5 

L  Manpower  0.2 

M.  Medical  4.6 

N.  Missile  Sqn  2.7 

O.  MWR  0.2 

P.  Personnel  13 

Q.  Public  Affairs  0.1 

R.  Safety  0.2 


S.  Security  Police  7.0 

T.  Services  1.2 

U.  Social  Actions  0.2 

V.  Supply  4.5 

W .  Transportation  3.4 

X.  Training  1.4 

Y.  Weather  0.2 

Z.  Other  203 
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Section  I  -  Demographics 

Please  do  not  proceed  until  you  have  completed  the  section  entitled  "Filling  In  The  Answer  Sheet" . 


L  Your  grade: 

e.  E-7  Master  Sergeant  41.8 

f.  E-8  Senior  Master  Sgt  1.5 

g.  E-9  Chief  Master  Sgt  0.1 

h.  G-S  Civilian  0 

i.  W-G  Civilian  0 


a.  E-3  Airman  First  Class  0 

b.  E-4  Senior  Airman/Sergeant  0.4 
c  E-5  Staff  Sergeant  1.2 

d.  E-6  Technical  Sergeant  54.9 


2.  Years  on  Active  Duty  (Military);  years  with  Civil  Service  (Civilians): 


a.  1  to  4  years  0 

b.  4  to  8  years  0.5 

c  8  to  12  years  11.4 

d.  12  to  16  years  40.0 

e.  Over  16  years  48.1 


99.5%  *  or  more 
88.1%  =  12  or  more 
48.1%  =  16  or  more 


3.  Your  current  role  in  the  training  process  is:  (Mark  all  that  apply) 

a.  Instructor  (Classroom,  does  not  include  OJT  Trainer)  10.2 

b.  OJT  Trainer  63.8 

c  Ancillary  Training  Trainer  9.4 

d.  Squadron  training  monitor  2.4 

e.  Section  training  monitor/coordinator  38.2 

f.  Squadron  OJT  monitor  1.1 
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Section  II  -  OJT  Trainer/Supervisor 


Complete  this  section  only  if  you  are  currently  designated  as  an  OJT  Trainer  on  a 
Trainee's  AF  Form  623  or  other  approved  form,  or  if  you  are  a  supervisor  with 
overall  OJT  responsibility  for  a  section.  If  you  are  not  an  OJT  Trainer/Supervisor, 
proceed  to  Section  m. 

(1155  (83.8%)  responded) 


4L  For  how  many  individuals  do  you  serve  as  OJT  trainer? 

a.  One  14.7 

b.  Two  15.7 
c  Three  12.7 

d.  More  than  three  39.1 

e.  OJT  supervisor  only  17.7 


5.  How  were  you  trained  or  qualified  as  an  OJT  trainer?  (Mark  all  that  apply) 

a.  ATC  Technical  Instructor  Course  8.0  (9.6) 

b.  OJT  trainer/Supervisors  Course  (FTD)  62.9  (75.2) 

c  Military  instructor  course  (TAC  Maintenance  Instructor  Course,  AIC,  etc.) 
3.3  (4.0) 

d.  Civilian  education /training  experience  9.5  (11.3) 

e.  Professional  Military  Education  (PME)  42.1  (50.2) 

f.  On-the-job  experience  only  (do  not  select  any  others  above)  17.8  (21.3) 


6.  Approximately  what  proportion  of  your  duty  time  is  normally  spent  in 
providing  and/or  supervising  on-the-job  training? 


a.  All  or  nearly  all  2.9 

b.  About  three-quarters  4.5 
c  About  half  15.2 


d.  About  one-quarter  25.9 

e.  About  5  to  10  percent  32  0 

f.  Less  than  5  percent  19.4 
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7.  How  long  have  you  been  an  OJT  trainer/supervisor? 

a.  One  year  or  less  7.1 

b.  One  to  three  years  7.5 

c.  Three  to  five  years  7.3 

&  Moie  than  five  years  78.1 

8.  When  entering  your  section,  a  person  receives  an  initial  training  evaluation  by: 

(Mark  all  that  apply) 

a.  Section  supervisor  52.4  (62.5) 

b.  Myself  as  OJT  Trainer  41.6  (49.6) 

c.  Another  OJT  Trainer  18.3  (21.8) 

d.  Section  training  coordinator/monitor  20.8  (24.8) 

e.  Squadron  OJT  monitor  14.1  (16.8) 

f.  None  of  the  above  3.2  (3.8) 


9.  During  the  initial  evaluation,  what  sources  are  used  to  determine  the  student’s 
training  needs:  (Mark  all  that  apply) 

a.  Initial  evaluations  not  usually  conducted  4.1  (4.8) 

b.  Previous  training  records  (AF  Form  623,  AF  Form  797,  etc.)  675  (80.6) 
c  Computer  products  (CAMS,  CASB,  MMICS,  APDS,  etc.)  34.0  (40.6) 

d.  Training  S’.rndards  (STS,  AFJQS,  CJQ5)  56.1  (66.9) 

e.  W^rkcenter  Master  Task  List  (MTL)  44.0  (52.6) 

f.  Interview  (discuss  student's  qualifications)  58.8  (70.2) 

g.  Over-the-shoulder  evaluation  20.9  (24.9) 

h.  Written  test  5.1  (6.1) 

i.  Trainer  assessment  26.1  (31.2) 

j.  Performance /Product  evaluation  24.6  (29.3) 


10.  How  are  task  evaluations  most  often  completed?  (Select  One) 

a.  Trainer/supervisor  over-the-shoulder  evaluation  63.5 

b.  Informal  (asking  questions  concerning  the  task)  22.2 

c  Outside  (third  party)  evaluation  (e.g.,  another  section/unit)  4.5 

d.  Outside  agency  evaluation  (e.g.  Quality  Assurance)  8.5 

e.  Paper  and  pencil  testing  1.3 
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Please  use  the  following  scale  to  rate  each  aspect  of  OJT  training  in  items  11-20. 

a.  Not  a  problem  at  present  —  no  improvement  needed 

b.  Only  minor  problems  with  this  at  present 

c.  Some  problems  with  this  —  improvement  desirable 

d.  Major  problems  with  this  —  improvement  urgently  needed 


11.  Availability  of  appropriate  written  training  materials  (such  as  training  packages, 
texts,  manuals,  tech  orders,  forms). 

a.  48.9 

b.  25.4 

c.  19.6 

d.  6.1 

12.  Availability,  when  needed,  of  training  facilities  and  equipment  such  as 
classrooms,  aircraft  and  mock-ups. 

a.  53.9 

b.  22.4 
c  17.3 
d.  6.4 

13.  Availability  of  equipment  for  presenting  individual  training,  such  as  com¬ 
puters,  VCRs,  and  simulators. 

a.  56.5 

b.  22.7 
c  15.6 
d.  5.2 

14.  Maintenance  and  repair  of  equipment  used  in  training,  such  as  mock-ups, 
computers,  VCRs,  IVDs,  or  simulators. 

a.  61.9 

b.  22.1 
c  11.2 
d.  4.7 
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15.  Having  enough  time  to  perform  my  training  duties. 

a.  37.0 

b.  28.6 
c  24.0 
d.  103 

16.  Trainees  available  for  required  training. 

a.  52.5 

b.  28.5 
c  13.8 
d.  5.2 

17.  Appropriateness  of  externally  produced  training  materials  for  rise  at  the  unit 
level. 

a.  48.6 

b.  26  2 
c  18.9 
d.  6.3 

18.  Integrating  job  skills  training  with  other  mission  requirements. 

a.  483 

b.  31.6 
c  15.1 
d.  5.1 

19.  Training  on  tasks  that  are  not  performed  very  often. 

a.  30.5 
h  37.1 
c  23.6 
d.  8.7 

20.  OJT  keeping  up  with  changes  in  technology  used  for  the  job. 

a.  403 
b  31.4 
c  203 
d.  8.0 
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Section  III  -  Training  Development 


This  section  applies  to  individuals  who  currently  plan  or  develop  training  courses 
or  exercises,  either  ancillary  or  job-related.  At  the  unit  level,  this  may  be  in  special 
job-related  training  programs  such  as  FTD  or  AMQP,  ancillary  training  such  as 
Self /Buddy  Care  or  CPR,  or  special  training  directed  by  Commander/ Supervisor 
such  as  introducing  new  equipment  or  policy. 

If  you  are  not  involved  in  the  development  of  training,  go  to  Section  IV. 


(588  (43%)  responded) 

21.  What  portion  of  your  duty  time,  overall,  is  spent  on  the  development  of 
training? 


a.  All  or  nearly  all  3.2 

b.  About  three-quarters  5.6 
c  About  half  9.2 


d.  About  one-quarter  22.3 

e.  About  5  to  10  percent  33.0 

f.  Less  than  5  percent  26.7 


I 

1 
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22.  What  types  of  training  do  you  develop  or  help  to  develop?  (Mark  all  that  apply) 

a.  Formal  courses  5.1  (12.1) 

b.  OJT  23.8  (55.8) 

c  Exercises  or  examples  to  supplement  formal  courses  12.4  (29.1) 

d.  Simulations  10.6  (24.8) 

e.  Field  exercises  12.2  (28.6) 

f.  Other  13.6  (32.0) 


23.  What  sources  do  you  use  to  determine  training  content?  (Mark  all  that  apply) 

a.  STS  27.4  (64.3) 

b.  AFJQS  11.9  (27.9) 

c  Command  JQS  10.1  (23.6) 

d.  Master  Task  List  22.1  (51.9) 

e.  Occupational  Survey  Report  1.4  (3.2) 

f.  Unit-level  task  analysis  8.6  (20.1) 

g.  MAJCOM  training  plans  5.5  (12.9) 

h.  Talking  to  functional  managers  about  need  for  training  8.4  (19.7) 

i.  Training  evaluation  reports  6.9  (16.2) 

j.  Content  determined  by  higher  level  or  headquarters  8.1  (19.0) 

k.  Other  11.8  (27.7) 

l.  Don't  know  0.8  (1.9) 

24.  About  what  portion  of  the  training  materials  used  in  your  training  activities  are 
developed  locally? 

a.  All  or  nearly  all  27.4 

b.  About  three  quarters  14.7 
c  About  half  23.9 

d.  About  one  quarter  13.1 

e.  About  5  to  10  percent  15.2 

f.  None  5.7 
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25.  Which  of  the  following  steps  do  you  usually  do  when  developing  training? 

(Mark  all  that  apply) 

a.  Conduct  a  training  needs  assessment  25.7  (60.4) 

b.  Select  tasks  requiring  training  31.1  (73.0) 
c  Develop  objectives  28.5  (66.8) 

d.  Develop  tests  18.2  (42.7) 

e.  Plan  sequence  of  instruction  24.1  (56.6) 

f.  Select  instructional  methods  21.9  (51.4) 

g.  Select  instructional  media  14.1  (33.2) 

h.  Determine  resource  and  funding  requirements  83  (19.4) 

i.  Develop  instructional  materials  19.4  (45.4) 
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Section  IV  -  Training  Delivery 


Training  delivery  refers  to  the  actual  presentation  of  a  training  class  or  course.  The 
training  might  be  formal  or  informal,  job-related  or  ancillary.  If  you  do  not  deliver 
training,  skip  to  Section  V. 

(793  (58%)  Responded) 


26.  Is  presentation  of  training  your: 

a.  Primary  duty  7.2 

b.  Part  of  your  primary  duty  52.9 
c  Assigned  additional  duty  27.8 
d.  Volunteer  additional  duty  12.1 


27.  What  portion  of  your  time,  overall,  is  normally  spent  in  delivering  training? 

a.  All  or  nearly  all  2.4  d.  About  one-quarter  24.5 

b.  About  three-quarters  3.7  e.  About  5  to  10  percent  35.8 

c  About  half  12.9  f.  Less  than  5  percent  20.8 

28-35  Please  use  this  set  of  responses  to  indicate  how  frequently  you  use  each  of  the 
following  methods  when  delivering  training: 

a.  Use  this  method  with  nearly  all  trainees 

b.  Use  method  with  about  half  of  trainees 

c  Use  method  with  about  a  quarter  of  trainees 

d.  Have  used  this  method,  but  rarely 

e.  Never  used  this  method 

(Choose  flue  response  for  EACH  item). 

28.  Hands-on  and  over-the-shoulder  instruction. 

a.  77.6  d.  3.0 

b.  9.0  e.  3.0 

c  7.3 
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Assign  self-paced  instruction  using  written  materials,  such  as  texts,  references  or 
manuals. 

a.  32.0  d.  27.6 

b.  10.4  e.  18.6 

c  11.4 


30.  Group  of  trainees  study  written  materials,  such  as  CDCs,  together. 

a.  9.0  d.  26.0 

b.  4.4  e.  54.2 

c  63 

31.  Computer-assisted  instruction  (CA1). 

a.  8.9  d.  17.9 

b.  33  e.  63.1 

c  6.7 

32.  Interactive  video  disks  (IVD). 

a.  23  d.  11.9 

b.  2.6  e.  78.8 

c  4.5 

33.  Video  cassette  recordings  (VCR). 

a.  14.4  d.  24.6 

b.  7.4  e.  44.6 

c  8.9 

34.  Simulators  or  mock-ups. 

a.  18.4  d.  18.2 

b.  6.8  e.  46.8 

c  9.8 

35.  Lecture/presentation  in  a  classroom  setting. 

a.  33.1  d.  20.6 

b.  13.2  e.  21.2 

c  12.0 
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36-43  Please  rate  each  of  the  following  teaching  methods  in  terms  of  how  effective 
the  method  is  for  helping  the  average  trainee  learn  the  tasks.  Use  the  following 
scale  in  rating  each  method: 

a.  Very  effective  -  trainees  learn  quickly  with  this  method 

b.  Effective 

c  Somewhat  effective 

d.  Somewhat  ineffective 

e.  Ineffective 

f.  Very  ineffective  -  trainees  learning  a  task  with  this  method  usually  ~'ust  be  re- 
taught 

g.  No  experience  with  method 
(Choose  one  response  for  each  item) 

36.  Hands-on  and  over-the-shoulder  instruction. 


a. 

73.4 

e. 

0.8 

b. 

20.2 

f. 

0.4 

c. 

4.0 

g- 

0.9 

d. 

0.2 

37.  Self-paced  instruction  using  written  materials,  such  as  texts,  references  or 
manuals. 

a.  6.5  e.  3.2 

b.  32.7  f.  2.1 

c  38.1  g.  9.3 

d.  8.0 

38.  Group  study  of  written  materials. 

a.  5.4  e.  6.4 

b.  21.0  f.  2.1 

c  29.1  g.  27.2 

d.  8.7 

39.  Computer-assisted  instruction  (CAI). 


a.  4.4  e.  3.1 

b.  14.5  f.  1.5 

c  16.0  g.  56.3 

d.  4.2 
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40.  Interactive  video  disks  (TVD). 

a.  23  e.  4.1 

b.  72  f.  13 

c  11.6  g.  69.7 

d.  3.6 

41.  Video  cassette  recordings  (VCR). 

a.  4.8  e.  3.9 

b.  20.7  f.  1.1 

c  27.1  g.  35.5 

d.  7.0 

42.  Simulators  or  mock-ups. 

a.  19.6  e.  2.8 

b.  23.4  f.  0.4 

c  133  g.  38.7 

d.  1.7 

43.  Lecture/presentation. 


a. 

11.3 

e. 

2.8 

b. 

35.4 

f. 

1.4 

c 

32.0 

g- 

11.6 

d. 

5.4 
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44-51  Please  rate  each  training  delivery  method  below  in  terms  of  its  reliability.  By 
reliability,  we  mean  whether  the  method  is  always  available  for  use,  whether 
instruction  is  consistent  using  the  method,  and  whether  breakdowns,  malfunctions 
or  non-availability  occur.  Please  use  the  following  scale  to  rate  each  method: 

a.  Very  reliable  -  no  problems  with  it 

b.  Reliable 

c.  Somewhat  reliable  -  some  problems  occur  with  it 

d.  Unreliable  -  problems  frequently  occur 

e.  Very  unreliable 

f.  Never  used  this  method 

(Choose  one  response  for  each  item) 

44.  Hands-on  and  over-th e-shoulder  instruction. 

a.  61.8  d.  .8 

b.  30.9  e.  .1 

c  5.3  f.  1.1 

45.  Self-paced  instruction  using  written  materials,  such  as  texts,  references  or 
manuals. 


a.  10.8 

d.  6.7 

b.  31.6 

e.  2.6 

c  38.9 

f.  9.3 

46.  Group  study  of  written  materials. 

a.  5.2 

d.  8.8 

b.  19.8 

e.  5.6 

c  33.4 

f.  27.2 

47.  Computer-assisted  instruction  (CAI), 

a.  4.1 

d.  4.7 

b.  12.2 

e.  3.8 

c  18.0 

f.  57.2 

48.  Interactive  video  disks  (IVD). 

a.  1.7 

d.  4.3 

b.  6.9 

e.  4.6 

c  12.8 

f.  69.7 
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49.  Video  cassette  recordings  (VCR). 

a.  5.9  d.  4.0 

b.  233  e.  3.1 

c  28.1  f.  35.7 

50.  Simulators  or  mock-ups. 

a.  12.7  d.  2.9 

b.  26.6  e.  2.4 

c  16.9  f.  38.5 

51.  Lecture/presentation. 


a. 

16.4 

d. 

4.0 

b. 

35.8 

e. 

1.7 

c 

303 

f. 

11.8 
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Section  V  -  Opinions  About  Training 


This  section  asks  for  your  own  opinions  about  some  aspects  of  training  methods  and 
training  delivery  at  your  base.  Please  use  the  following  scale: 

a.  Strongly  agree 

b.  Agree 

c  Disagree 

d.  Strongly  disagree 

e.  Not  applicable  or  don’t  know 


52.  I  feel  confident  that  I  am  technically  qualified  to  be  a  trainer. 


a.  56.5 

b.  38.1 
c  3  3 


d.  0.8 

e.  1.2 


53.  I  feel  confident  in  my  teaching  skills. 


a.  51.1 

b.  45.0 

c  2.6 


d.  0.4 

e.  1.0 


54.  The  training  media/methods  available  today  are  adequate  for  my  needs. 


a.  15.3 

b.  603 

c.  17.1 


d.  4.7 

e.  2.7 


55.  Training  needs  that  I  deal  with  are  continually  changing,  so  it  is  difficult  to  plan 
for  the  future. 


a.  10.3 

b.  28.5 
c  483 


d.  8.3 

e.  4.4 


56.  Standards  for  training  in  my  unit  go  far  beyond  general  Air  Force  standards. 


a.  10.8 

b.  29.7 
c  44.8 


d.  9.4 

e.  5.4 
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57.  When  technologies  for  training  delivery  (e.g.,  automated  systems)  are 
introduced  in  our  unit,  they  are  used  to  full  capacity. 

a.  7.0  d.  7.5 

b.  35.1  e.  23.7 

c  26.6 

58.  For  the  jobs  in  my  unit,  hands-on  training  is  generally  more  effective  than 
classroom  training. 

a.  52.0  d.  1.0 

b.  38.8  e.  1.7 

c  6.4 

59.  Computer-assisted  training  technologies  can  be  as  effective  as  classroom 
training. 

a.  11.1  d.  4.1 

b.  41.0  e.  23.8 

c  19.9 

60.  This  unit  has  sufficient  resources  to  provide  more  training  thar  :t  is  currently 
conducting. 

a.  9.7  d.  10.7 

b.  27.9  e.  11.2 

c  40.5 

61.  Training  packages  developed  outside  the  unit  (for  example,  at  headquarters 
level  or  by  contractors)  are  generally  useful  for  squadron  level  training. 

a.  33  d.  7.7 

b.  45.1  e.  19.9 

c  24.0 

62.  Contractors'  technical  representatives  fill  a  crucial  role  in  the  accomplishment 
of  technical  training  requirements  within  my  squadron. 

a.  4.6  d.  13.9 

b.  15.0  e.  38.8 

c  27.8 
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63.  Training  devices  and  simulators  are  used  frequently  for  training  in  my  unit. 

a.  73  d.  12.4 

b.  28.9  e.  22.9 

c  2 83 
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Section  VI  -  Future  Training 


The  following  section  describes  a  possible  future  situation. 

At  some  time  in  the  future  the  Air  Force  directs  that,  for  most  AFSCs,  time 
available  for  formal  technical  school  training  will  be  reduced  by  half.  Substantial 
training  responsibilities  are  to  be  shifted  to  the  base  and  squadron  level.  In  order  to 
accomplish  this  new  tasking,  base  and  squadron  training  and  training  management 
capabilities  need  to  be  enhanced. 

Answer  the  following  questions  assuming  that  YOU  have  been  tasked  to  help  deter¬ 
mine  requirements  at  your  base  for  supporting  increased  training  responsibilities 
As  usual,  these  increased  responsibilities  must  be  supported  with  as  small  an 
increase  in  training  manpower  as  possible.  Do  not  base  your  responses  on  current 
regulations,  operating  policies,  or  capabilities. 


64-66  As  part  of  the  restructuring  of  training,  Air  Force  has  directed  base  and 
squadron  level  training  organizations  to  develop  in-house  capabilities.  Please  use 
the  following  list  to  answer  items  64-66. 

a.  Develop  training  objectives 

b.  Develop  knowledge  tests 
c  Develop  performance  tests 

d.  Plan  sequence  of  instruction 

e.  Select  instructional  methods 

f.  Select  instructional  media 

g.  Determine  resource  and  funding  requirements 

h.  Develop  instructional  materials 

i.  Validate  the  approach  and  content  used 

j.  Do  not  know 

64.  If  your  unit  has  been  directed  to  perform  all  the  above  tasks,  which  of  the  tasks 
listed  above  would  be  completely  new  responsibilities  to  the  unit?  (Mark  all 
that  apply) 


a. 

13.8 

f. 

19.8 

b. 

25.6 

g- 

39.7 

c 

20.4 

h. 

25.7 

d. 

16.0 

i. 

32.1 

e. 

14.7 

j- 

29.5 
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65.  Which  one  of  the  tasks,  given  above  would  be  the  single  most  difficult  to 

implement  without  additional  manpower,  training,  or  equipment?  (Select  one) 


a. 

2.7 

f. 

3.7 

b. 

3.6 

& 

19.3 

c. 

3.4 

h. 

24.4 

d. 

2.0 

i. 

9.9 

e. 

13 

j- 

29.1 

66.  For  each  of  the  tasks  listed,  mark  the  ones  which  would  require  training 
personnel  to  receive  additional  training  to  perform.  (Mark  all  that  apply) 


a. 

20.2 

f. 

21.9 

b. 

273 

g- 

43.7 

c 

253 

h. 

36.8 

d. 

20.8 

i. 

36.4 

e. 

19.1 

j- 

25.0 

67-68  As  part  of  the  new  training  strategy.  Air  Force  has  directed  that  OJT  be  the 
primary  method  of  job  skills  training  beyond  technical  school.  You  have  been 
tasked  to  provide  additional  training  to  all  OJT  instructors  on  teaching  and  as¬ 
sessment  methods  in  the  work  environment 


67.  If  you  were  free  to  select  any  means  of  instructional  delivery  to  teach  OJT  instruc¬ 
tors  (train  the  trainers),  which  of  the  following  would  you  believe  to  be  most 
effifcieal  in  terms  of  scheduling,  record  keeping,  and  other  administrative  tasks? 
(Select  one) 

a.  Instructor-led  classroom  instruction  62.3 

b.  Videotaped  "lectures"  7.6 

c  Computer-based  training  22.2 
d.  Self  study  printed  materials  8.0 


68.  Which  of  the  following  means  of  instructional  delivery  would  you  feel  to  be  the 
moat  effective  for  preparing  OJT  instructors  to  carry  out  their  duties?  (Select 
one) 

a.  Instructor-led  classroom  instruction  77.1 

b.  Videotaped  "lectures"  6.2 

c  Computer-based  training  113 
d.  Self  study  printed  materials  5.4 
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69-72  In  support  of  the  increased  requirements  for  base  and  squadron  level  training, 
you  are  tasked  to  prepare  a  plan  to  administer  and  conduct  training.  Please  use  the 
following  list  to  answer  il  ;ms  69-72. 

a.  Scheduling  students 

b.  Scheduling  resources 

c  Tracking  student  progress 

d.  Managing  training  requirements 

e.  Maintaining  instructional  materials  and  equipment 

f.  Updating  instructional  materials  and  equipment 

g.  Do  not  know 


69.  For  which  of  the  tasks  listed  above  would  you  propose  to  continue  to  use 

current  methods  and  procedures  with  little  or  no  modification?  (Mark  all  that 

apply) 


a. 

47.7 

e. 

32.0 

b. 

34.1 

f. 

23.6 

c 

40.9 

& 

23.6 

d. 

34.7 

70.  For  each  of  the  tasks  listed  above,  mark  the  ones  which  would  require  training 
personnel  to  receive  additional  training  to  perform.  (Mark  all  that  apply) 


a. 

9.9 

e. 

31.8 

b. 

18.7 

f. 

48.4 

c. 

20.7 

& 

26.4 

d. 

36.5 

71.  For  which  single  task  from  the  above  list  would  automated  record  keeping 
prove  to  be  most  beneficial,  given  increased  base  and  unit  training 
requirements?  (Select  one) 


a. 

15.3 

e. 

4.2 

b. 

4.3 

f. 

8.4 

c. 

33.8 

g- 

15.9 

d. 

18.1 
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72.  For  which  single  task  from  the  above  list  would  you  suggest  that  additional 
manpower  would  be  needed  regardless  of  the  level  of  automation  provided? 
(Select  one) 


a. 

2.9 

e. 

20.0 

b. 

3.0 

f. 

27.0 

c 

65 

& 

23.8 

d. 

16.9 

73-75  In  support  of  the  increased  requirements  for  base  and  squadron  level  training, 
evaluation  responsibilities  will  be  substantially  increased.  Please  use  the  following 
list  to  answer  questions  73-75. 

a.  Evaluating  student  performance  during  training. 

b.  Evaluating  student  performance  on  completion  of  training, 
c  Evaluating  job  performance  during  training. 

d.  Evaluating  job  performance  in  relation  to  training  performance. 

e.  Measuring  the  usefulness  and  cost-benefits  of  various  local  training  programs. 

f.  None  of  the  above. 

73.  Which  one  of  the  evaluation  responsibilities  listed  do  you  believe  would  prove 
to  be  the  most  difficult  to  implement  at  the  base  or  squadron  level?  (Select  one) 

a.  3.2  d.  7.7 

b.  3.3  e.  66.0 

c  3.0  f.  16.8 

74.  For  which  of  the  tasks  listed  would  you  propose  to  continue  to  use  current 
methods  and  procedures  with  little  or  no  modification?  (Mark  all  that  apply) 

a.  56.9  d.  32.8 

b.  50.0  e.  8.3 

c  48.4  f.  15.5 

75.  For  which  single  task  would  automated  record  keeping  prove  to  be  most 
beneficial,  given  increased  base  and  unit  training  requirements?  (Select  one) 

a.  9.0  d.  11.9 

b.  11.4  e.  35.0 

c  4.4  f.  28.3 


153 


Survey  for  Trainers 
Section  VII 


Section  VII-  Your  Training  Activities  During 
Desert  Shield/Desert  Storm 


We  need  to  know  more  about  how  to  plan  for  unit-level  training  in  a  wartime 
environment.  Please  answer  this  section  in  terms  of  your  unit's  training  during  the 
period  of  the  conflict  with  Iraq,  from  August,  1990  to  February,  1991.  Answer  this 
section  whether  or  not  vou  were  deployed  to  the  Middle  East. 

76.  In  support  of  the  conflict  with  Iraq,  were  you:  (Mark  all  that  apply) 

a.  Deployed  to  the  Middle  East  14.2 

b.  Deployed  from  CONUS  to  overseas,  but  to  a  location  other  than  Middle  East.  4.6 
c  Deployed  from  overseas  to  CONUS.  0.S 

d.  Deployed  within  the  U.S.  1.8 

e.  Did  not  deploy  -  remained  in  current  location.  78.4 


77.  If  you  deployed  in  support  of  the  conflict,  was  this  deployment  with  your  entire 
unit,  part  of  your  unit,  or  individually? 

a.  Does  not  apply  -  did  not  deploy.  79.9 

b.  Individually,  not  with  my  unit.  3.8 
c  With  part  of  my  unit.  14.0 

d.  With  entire  unit.  2.3 


78.  During  the  period  of  the  conflict  (from  August  1990  to  February  1991)  which  of 
the  following  types  of  training  did  you  provide?  (Mark  all  that  apply) 

a.  Normal  OJT  trainiir  53.9 

b.  Special  OJT  to  opera- e/maintain  equipment  in  the  desert  environment.  7.9 
c  Special  unit  training  to  learn  to  live  in  the  desert  environment.  5.3 

d.  Additional  ancillary  training  for  threats  such  as  nudear/chemical/biological 
warfare.  15.7 

e.  Training  for  working  and  living  within  an  Arabic  culture.  3.5 

f.  Normal  ancillary  training.  22.0 

g.  Other  training.  18.9 

h.  None.  24.4 
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79.  During  the  period  of  conflict/  the  time  that  was  devoted  to  training: 

a.  Increased  substantially  10.6 

b.  Increased  slightly  16.0 
c  Did  not  change  40.7 

d.  Decreased  slightly  9.2 

e.  Decreased  substantially  13.5 

f.  Stopped.  No  training  given  during  this  period.  9.9 


80-87  During  Desert  Shield/Storm/  what  changes  occurred  in  your  unit's  training? 
Please  use  the  following  responses  to  indicate  what  types  of  changes  occurred  in 
each  aspect  of  training  listed  in  items  80  to  87.  (Mark  all  that  apply  for  each  item) 

a.  Decreased 

b.  Increased 

c  Waived  part  or  all  of  normal  activities 

d.  Postponed  some  or  all 

e.  Changed  method  for  doing  this 

f.  Do  not  normally  perform  this  training 

g.  Not  applicable;  did  not  change 


80.  Formal  job  related  training  (such  as  classroom  instruction). 


a. 

19.4 

e. 

2.2 

b. 

7.5 

f. 

10.3 

c 

6.6 

g- 

47.1 

d. 

13.1 

81.  Informal  job  related  training  (such  as  unit  meetings/  word  of  mouth/  etc). 


a.  10.0 

e. 

2.0 

b.  28.6 

f. 

3.7 

c  3.0 
d  5.6 

& 

46.8 

82.  CDCs. 

a.  7.5 

e. 

1.1 

b.  2.1 

f. 

5.3 

c  4.0 
d  9.5 

g- 

69.0 
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83.  Ancillary  training. 

a.  10.4  e.  2.2 

b.  17.8  f.  4.5 

c.  5.1  g.  50.1 

d  10.2 

84.  Documentation  of  training. 

a.  10.7  e.  2.4 

b.  6.7  f.  2.5 

c  4.6  g.  66.0 

d  5.7 

85.  Use  of  automated  training  delivery  systems. 

a.  7.9  e.  1.5 

b.  4.4  f.  15.4 

c  3.0  g.  62.7 

d  4.0 

86.  Direct  combat-related  training. 

a.  3.7  e.  1.4 

b.  27.7  f.  14.6 

c  1.5  g.  47.1 

d  25 


87.  Evaluation  of  training. 


a. 

12.1 

e. 

3.7 

b. 

9.1 

f. 

2.4 

c. 

4.8 

& 

60.8 

d 

5.0 
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88.  How  well  trained  were  you  for  your  duties  during  Desert  Storm  (i.e.  were  you 
trained  on  the  right  things)? 

a.  Well  trained  57.1 

b.  Somewhat  well  trained  26.3 
c  Little  useful  training  5.9 

d.  No  useful  training  10.7 

Thank  you  for  completing  this  survey.  Please  return  this  survey  form  and  your 
answer  sheet  to  the  person  who  gave  them  to  you. 
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Trainee 
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Squadron  Level  Training 
Survey  Form 

For  Recipients  of  Training 


Prepared  For:  Technical  Training  Research  Division 

Human  Resources  Directorate  of  the 
Armstrong  Laboratory 
Brooks  A.F.B.,  TX  78235-5601 


Prepared  By:  HAY  Systems,  Inc. 

Suite  650 

2000  M  Street,  N.W. 
Washington,  DC  20036 


1  November  1991 


Version  Number: 

0004 
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To  All  Survey  Participants: 


You  have  been  selected  to  take  part  in  a  survey  covering  various  aspects  of  training 
in  the  Air  Force.  This  project  is  being  sponsored  by  the  Technical  Training  Research 
Division,  Human  Resources  Directorate  of  the  Armstrong  Laboratory,  Brooks  AFB, 
TX. 

The  information  will  be  used  for  research  on  training  at  Operational  Units  in  the 
Air  Force.  Rapidly  changing  technology  in  a  changing  world  requires  that  Air  Force 
training  be  responsive.  Your  assistance  with  this  survey  will  help  us  better  un¬ 
derstand  the  viewpoint  of  those  involved  in  base  level  training  and  the  on-site 
training  process. 

The  information  you  provide  will  be  kept  in  strictest  confidence.  The  background 
information  that  you  are  asked  to  provide  on  the  answer  sheet  will  be  used  to  help 
assure  that,  overall,  the  people  completing  this  survey  are  representative  of  the  total 
population  in  the  Air  Force. 


Privacy  Act  Statement 

Authority:  5  USC  Sec  301,  EQ9397,  and  AFR  35-2 

Principal  Purpose:  Collection  of  information  concerning  training  program 
development  and  evaluation.  SSAN  for  positive  identifications. 

Routine  Area:  Personnel  research  and  personnel  management  system  applications. 

Disclosure:  Failure  to  complete  this  requirement  will  detract  from  the  Air  Force's 
capability  to  carry  out  programs  listed  above. 
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Squadron  Level 
Training  Survey  Form 


'  n  -N  '  ~  ■>*'  m  ;•  V>>  ‘i  *  ■  <  '  "  v  #  x  v  *>  '  >•  L*  ,'s'  > 

Instructions  For  Completing  The  Questionnaire  ; 


•  You  should  have  two  sharpened  pencils  and  an  eraser. 

•  Mark  all  answers  on  the  answer  sheet  only. 


Do  Not  Mark  On  The  Survey  Forms 


Wrong 


Wrong 

(a  1  C  (B)  )  C  c 
Right 

t  a  j  flm  ct 


•  Make  heavy  black  marks  that  fill  the  oval  completely.  Wrong; 

(See  examples  at  right,  which  show  both  correctly  and  °  _  _ 

incorrectly  marked  answer  spaces.)  '-/& — '  ' — § — J  Q— £-J 

Wrong 

•  Fill  in  just  one  oval  unless  you  are  given  instructions  to  CEUD  C  ,9  )  La/'j 

mark  all  that  apply.  _ Wrong  _ 

CUD  GO  rm 

•  If  you  wish  to  change  an  answer,  erase  completely.  Right 

•  Make  no  stray  marks  on  the  answer  sheet. 


Filling  In  The  Answer  Sheet 


Do  Not  Fill  In  Your  Name  or  Date  of  Birth 

Before  answering  any  of  the  survey  questions,  fill  in  the  following  information  in  the  spaces  as  indicated. 
Name  Grid: 

•  Using  LIST  A  on  the  following  page,  enter  the  letter  that  matches  the  major  command,  held  operating 
agency,  or  direct  reporting  unit  in  the  first  column  of  the  Last  Name  section  of  your  answer  sheet 

•  Using  LIST  B  on  the  following  page,  enter  the  tetter  that  matches  the  type  of  unit  to  which  you  are 
assigned  in  the  first  column  of  the  First  Name  section  of  your  answer  sheet 

Social  Security  Number  Grid: 

•  Fill  in  your  Social  Security  Number. 

"Grid"  Grid: 

•  Fall  in  the  Version  Number  located  in  the  lower  right  comer  of  the  Survey  form  cover  sheet;  use  all  four 
columns,  for  example,  0004. 

Be  sure  to  fill  in  the  information  as  shown  in  the  Grid-Marking  Example  shown  on  the  lower  right  hand 
'•omer  of  side  one  of  your  answer  sheet. 

Starting  in  row  1  of  your  answer  sheet  fill  in  your  responses.  This  survey  should  take  you  less  than  one  hour 
to  complete. 

After  completion  of  this  survey,  return  the  survey  form  and  the  answer  sheet  to  the  person  who  gave  you 
these  materials. 
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LIST  A 


AIR  FORCE  MAJOR  COMMANDS,  HELD  OPERATING  AGENCIES,  AND  DIRECT 

REPORTING  UNITS 

Enter  the  LETTER  that  matches  your  selection  on  the  first  Column  of  the  Last  Name 
section  of  your  answer  sheet. 


A.  Air  Force  Logistics  Command 
(AFLC)  2.7 

B.  Air  Force  Systems  Command 
(AFSC)  5.0 

C  Air  Force  Space  Command 
(AFSPACECOM)  0.1 

D.  Air  Training  Command  (ATC)  1.5 

E  Air  University  (AU)  1.5 

F.  Electronic  Security  Command  (ESC) 
3.5 

G.  Military  Airlift  Command  (MAC) 
14.4 

H.  Strategic  Air  Command  (SAC)  21.1 

I.  Tactical  Air  Command  (TAC)  15.2 

J.  Air  Force  Combat  Operations  Staff 
(AFCOS)  0.1 

K.  Air  Force  Communications  Agency 
(AFCA)  2.1 

L  Defense  Agency  0.0 

M.  Air  Force  District  of  Washington 
(AFDW)  0.3 

N.  Air  Force  Inspection  Center  (AFIC) 

0.2 

O.  Air  Force  Intelligence  Agency 
(AFLA)  0.2 


P.  Air  Force  Legal  Services  Center 
(AFLSO  0.1 

Q.  Air  Force  Management  Engineering 
Agency  (AFMEA)  0.1 

R.  Air  Force  Military  Personnel  Center 
(AFMPC)  0.1 

S.  Air  Force  Morale,  Welfare,  and 
Recreation  Agency  (AFMWRA)  0.1 

T.  Air  Force  Office  of  Medical  Support 
(AFOMS)  0.3 

U.  Air  Force  Office  of  Security  Police 
(AFOSP)  13 

V.  Air  Force  Office  of  Special  Investiga¬ 
tions  (AFOSD  0.0 

W.  Air  Force  Operational  Test  and 
Evaluation  Center  (AFOTEC)  0.2 

X.  Air  Force  Safety  Center  0.1 

Y.  US  Air  Force  Academy  (USAFA)  0.1 

Z.  Other  29.5 

(USAFE  21.5) 

(PACAF  8.3) 
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LISTB 


DESCRIPTION  OF  UNIT  OF  ASSIGNMENT 

Enter  the  LETTER  that  matches  your  selection  on  the  first  Column  of  the  First 
Name  section  of  your  answer  sheet. 


A.  Aircraft  Maint  20.9 

B.  Audiovisual  0.4 
G  Audit  0.1 

D.  Chaplain  0.3 

E.  Civil  Engnrg  6.8 

F.  Communications 

11.0 

G.  Comptroller  1.1 

H.  Education  0.8 

I.  Flying  Sqn  2.8 

J.  InfoMgmt  2.0 

K.  Legal  0.1 


L.  Manpower  0.0 

M.  Medical  4.7 

N.  Missile  Sqn  2.1 

O.  MWR  0.3 

P.  Personnel  1.4 

Q.  Public  Affairs  0.0 

R.  Safety  0.2 

S.  Security  Police  7.3 

T.  Services  1.5 

U.  Social  Actions  0.1 

V.  Supply  7.0 

W .  Transportation  5.1 
X  Training  0.6 

Weather  0.2 
Z.  Other  22.3 
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Section  I  -  Demographics 

Please  do  not  proceed  until  you  have  completed  the  section  entitled  "Filling  In  The  Answer  Sheet". 

l.Your  grade: 

a.  E-l  Airman  Basic  0.1 

b.  E-2  Airman  0.1 
c  E-3  Airman  First  Class  16.9 


2.  Years  on  Active  Duty? 

a.  less  than  one  0.3 

b.  1  to  2  years  5.5 
c  2  to  3  years  13.7 

d.  3  to  4  years  10.6 

e.  4  or  More  69.9 


3.  Are  you: 

a.  Female  15.6 

b.  Male  84.4 


4.  Racial  >  ethnic  background: 

a.  Afro-American /Black  16.7 

b.  Caucasian/ White  72.1 

c  Asian  or  Pacific  Islander  3.3 

d.  Hispanic  (regardless  of  race)  6.0 

e.  Other  1.8 


5.  What  is  the  highest  level  of  education  you  have  completed? 

a.  Less  than  high  school  0.3 

b.  High  school  29.4 

c  Some  college  but  less  than  2  years  49.1 

d. Two  or  mere  years  of  college  11.7 

e.  Associates  degree  5.9 

f.  Bachelors  degree  3.2 

g.  Advanced  degree  0.4 


d. E-4  Senior  Airman/Sergeant 

40.6 

e.  E-5  Staff  Sergeant  39.5 

f.  E-6  Technical  Sergeant  2.9 
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Section  II  -  Methods  Of  Training 


This  section  will  ask  you  questions  on  methods  of  training  used  at  your  duty 
location,  such  as  lecture,  demonstrations,  computer-based  instruction,  etc.  For  most 
of  these  questions  focus  your  attention  on  HOW  tilings  were  taught,  not  what  was 
taught. 

6.  Within  the  last  5  years  (or  since  you  entered  the  Air  Force,  if  less  than  5  years  ago) 
which  of  the  following  types  of  training  methods  have  you  experienced  at  your  duty 
location?  (Mark  all  that  apply) 

a.  Classroom  T/ith  an  instructor.  78.0 

b.  OJT  (on-job-training)  with  an  instructor.  56.2 
c  OJT  with  your  supervisor.  80.7 

d.  OJT  on  your  own,  using  reference  manuals,  books,  and  other  written 
materials.  80.6 

e.  Individual  training  using  computer-based  instruction  (CBI).  24.7 

f.  Individual  training  using  interactive  video  disk  (IVD).  9.2 

g.  Individual  training  using  video  cassette  recorder  (VCR).  30.3 

h.  Individual  training  using  a  simulator  or  other  individual  training  device. 
18.3 

i.  Correspondence  courses.  38.8 

j.  Contractor-conducted  training.  14.4 

7-16  Based  on  your  most  recent  experience  with  each  method  listed  in  items  7-16, 
use  the  following  responses  to  rate  how  useful  that  training  was  to  your  job. 

a.  Very  useful 

b.  Moderately  useful 
c  Moderately  useless 

d.  Very  useless 

e.  Did  not  experience  this  method  of  training  or  do  not  know 
(Choose  one  response  for  each  item) 

7.  Classes  with  an  instructor. 

a.  40.2  d.  2.3 

b.  43.2  e.  8.7 

c  5.6 
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8.  OJT  with  an  instructor. 

a.  41.1  d.  1.5 

b.  24.8  e.  29.3 

c  33 

9.  OJT  with  your  supervisor. 

a.  54.7  d.  2.0 

b.  293  e.  8.5 

c  5.4 

10.  OJT  on  your  own,  using  reference  manuals,  books,  and  other  written  materials. 

a.  39.6  d.  1.8 

b.  45.1  e.  7.0 

c.  6.4 

11.  Individual  training  using  computer-based  instruction  (CBI). 

a.  7.9  d.  2.7 

b.  16.6  e.  64.6 

c  8.2 

12.  Individual  training  using  interactive  video  disk  (IVD). 

a.  3.1  d.  2.6 

b.  7.0  e.  82.1 

c.  5.2 

13.  Individual  training  using  video  cassette  recorder  (VCR). 

a.  6.2  d.  4.8 

b.  19.4  e.  59.0 

c.  10.7 

14.  Individual  training  using  a  simulator  or  other  individual  training  device. 

a.  12.9  d.  2.0 

b.  12.8  e.  68.7 

c.  3.6 
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15.  Correspondence  courses. 

a.  122  d.  3.5 

b.  265  e.  47.0 

c  10.8 


16.  Contractor-conducted  training. 

a.  83  d.  1.8 

b.  9.9  e.  75.8 

c  4  2 


17-25  For  each  method  of  training  listed  in  items  17  to  25,  please  use  these  responses 
to  indicate  whether  you  would  prefer  more  or  less  of  this  method  of  training  in  the 
future. 


a.  Prefer  much  less  of  this  method  of  training. 

b.  Prefer  slightly  less  of  this  method  of  training. 

c.  Present  use  of  this  method  is  about  right. 

d.  Prefer  slightly  more  of  this  method  of  training. 

e.  Prefer  much  more  of  this  method  of  training. 

f.  Did  not  experience  this  method,  or  do  not  know. 

(Choose  one  response  for  each  item) 

17.  Classroom  with  an  instructor. 

a.  5.5  d.  17.0 

b.  11.5  e.  19.5 

c  41.3  f.  5.3 

18.  OJT  with  an  instructor. 

a.  2.2  d.  23.1 

b.  4.6  e.  26.5 

c  26.8  f.  17.0 


19.  OJT  with  your  supervisor. 

a.  2.5  d.  25.0 

b.  4.9  e.  27.6 

c  35.2  f.  4.7 
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20.  OJT  on  your  own,  using  reference  manuals,  books,  and  other  written  materials. 


a.  7.0 

b.  12.7 
c  453 


d.  18.0 

e.  11.6 

f.  5.4 


21.  Individual  training  using  computer-based  instruction  (CBI). 


a.  4.4 

b.  5.1 
c  123 


d.  9.4 

e.  14.7 

f.  53.9 


22.  Individual  training  using  interactive  video  disk  (IVD). 


a.  3.9 

b.  4.7 
c  7.1 


d.  4.6 

e.  113 

f.  68.1 


23.  Individual  training  using  video  cassette  recorder  (VCR). 


a.  6.0 

b.  83 
c  17.4 


d.  9.0 

e.  10.4 

f.  48.6 


24.  Individual  training  using  a  simulator  or  other  individual  training  device. 


a.  2.1 

d.  10.0 

b.  3.7 

e.  16.0 

c  10.8 

f.  57.3 

Correspondence  courses. 

a.  6.0 

d.  8.7 

b.  9.0 

e.  11.2 

c  27.4 

f.  37.7 
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Section  in  -  Upgrade  Training 


Complete  this  section  concerning  your  most  recent  upgrade  training  (UGT)  to  the  3, 
5,  or  7  skill  level. 

26.  What  is  your  current  skill  level  in  your  primary  AFSC? 

a.  Entry  level  0.3 

b.  3-level  5.4 
c  5-level  56.7 
&  7-level  37.5 


27.  Your  current  Upgrade  Training  (UGT)  status  is: 

a.  Currently  enrolled  13.8 

b.  Completed  within  past  6  months  9.2 

c  Completed  more  than  6  months  ago  77.0 


28.  For  your  most  recent  UGT,  how  many  trainers  have  you  been  assigned  to? 

a.  None  18.7 

b.  One  trainer  39.5 

c.  Two  trainers  19.3 

d.  Three  trainers  7.2 

e.  More  than  three  trainers  15.2 


29.  Upon  your  most  recent  UGT,  who  provided  an  initial  evaluation  to  determine 
your  training  requirements?  (Mark  all  that  apply) 

a.  Trainer  42.9 

b.  Section  supervisor  43.9 

c  Section  training  coordinator/monitor  14.3 

d.  Squadron  OJT  monitor  13.2 

e.  Someone  other  than  the  above  8.9 

f.  Did  not  receive  an  initial  evaluation  13.0 
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30.  How  are  task  certifications  most  often  completed  in  your  upgrade  training? 
fChnnse  one  response) 

a.  Trainer/supervisor  over-the-shoulder  evaluation  52.9 

b.  Trainer/supervisor  does  it  without  performance  evaluation  14.2 

c.  Informal  (questioning  concerning  the  tasks)  19.8 

&  Outside  third  party  evaluation  (e.g.,  by  another  section/unit)  6.3 
e.  Outside  agency  evaluation  (e.g..  Quality  Assurance)  6.7 


31.  How  much  of  your  duty  time  during  a  typical  week  is  devoted  to  receiving  OJT? 

a.  All  or  nearly  all  4.8 

b.  About  three-quarters  5.2 
c  About  half  10.4 

d.  About  one-quarter  18.5 

e.  About  5  to  10  percent  39.0 

f.  None  22.1 


32.  Is  your  time  for  OJT  included  in  your  work  schedule? 

a.  Yes  45.7 

b.  Sometimes  31.8 
c  No  22.5 


33-35  For  each  of  the  statements  in  items  33  to  35,  please  use  the  following  scale  to 
give  your  own  opinions  about  OJT. 

a.  Strongly  agree 

b.  Agree 

c  Disagree 

d.  Strongly  disagree 

e.  Do  not  know 

(Chpps£..pne responseJor  -gacfLSlatement) 

33.  There  is  sufficient  time  in  the  work  week  for  both  your  primary  duties  and  OJT. 


a.  23.0 

b.  46.5 
c  16.1 


d.  12.1 

e.  2.3 
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34.  The  initial  evaluation  (for  your  current  level)  was  an  accurate  assessment  of 
your  skills. 

a.  17.0  d.  5.9 

b.  57.2  e.  8.2 

c  11.6 

35.  Equipment  and  facilities  needed  for  OJT  are  usually  available. 

a.  14.8  d.  7.5 

b.  60.4  e.  2.6 

c  14.7 
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39-47  For  each  of  the  subjects  listed  in  item  39-47  indicate  which  type  of  teaching 
materials  and  methods  were  used.  Mark  all  methods  used  for  each  course,  in  your 
most  recent  attendance  at  each  course. 


a.  Lecture /presentation 

b.  Hands-on  use  of  real  equipment 
c  Slides/overhead  projectors 

d.  Use  of  simulators  or  mock-ups 

e.  Instructor  demonstrated  using  equipment,  while  trainees  watched 

f.  Computer-based  instruction  (CBI) 

g.  Interactive  video  disk  (TVD)-(Computer  and  video  disk) 

h.  Video  cassette  recorder  (VCR) 

i.  I  have  not  had  this  type  of  ancillary  training 


39.  Firearms  Training 


a.  67.7 

d.  14.8 

g- 

0.9 

b.  76.7 

e.  57.3 

h. 

6.4 

c.  16.0 

f.  0.6 

i. 

17.2 

40.  Base  Supply  Customer  Training 

a.  23.9 

d.  1.6 

g- 

0.6 

b.  5.5 

e.  3.7 

h. 

1.5 

c  13.3 

f.  0.4 

i. 

68.6 

41.  Self-Aid  and  Buddy  Care 

a.  71.6 

d.  28.3 

g- 

1.4 

b.  48.2 

e.  43.8 

h. 

31.7 

c  43.7 

f.  0.4 

i. 

10.3 

42.  CPR 

a.  54.6 

d.  35.4 

& 

0.9 

b.  41.1 

e.  41.5 

h. 

26.9 

c  24.2 

f.  0.4 

i. 

32.7 

43.  Chemical  Warfare  Defense  Training  for  High  Threat  Areas 

a.  69.0 

d.  28.4 

& 

0.9 

b.  64.5 

e.  48.6 

h. 

25.6 

c  36.6 

f.  0.7 

i. 

18.3 
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44.  Local  Conditions  Course 


a.  37.1 

d.  2.1 

g- 

0.4 

b.  4.6 

e.  2.9 

h. 

92 

c  19.4 

f.  0.1 

i. 

52.6 

Uniform  Code  of  Military  Justice  (UCMJ) 

a.  48.1 

d.  G.9 

& 

03 

b.  23 

e.  13 

h. 

9.1 

c  143 

f.  0.4 

i. 

43.1 

Communications  Security  (COMSEC) 

a.  683 

d.  13 

g- 

1.6 

b.  42 

e.  2.0 

h. 

16.7 

c  26.9 

f.  2.3 

i. 

123 

47.  Operations  Security  (OPSEC) 


a. 

67.1 

d. 

1.1 

g- 

13 

b. 

42 

e. 

2.7 

h. 

17.0 

c 

26.7 

f. 

23 

i. 

14.0 
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Section  V  -  Your  Training  Activities  During 
Desert  Shield/Desert  Storm 


We  need  to  know  more  about  how  to  plan  for  unit-level  training  during  a  wartime 
environment.  Please  answer  this  section  in  terms  of  your  unit's  training  during  the 
period  of  the  conflict  with  Iraq,  from  August,  1990  to  February,  1991.  If  you  were  in 
the  Air  Force  during  this  period  answer  this  section  whether  or  not  you  were 
deployed  to  the  Middle  East .  If  you  were  not  in  the  Air  Force  during  this  period 
leave  this  section  blank. 

48,  In  support  of  the  conflict  with  Iraq,  were  you: 

a.  Deployed  to  the  Middle  East.  13.5 

b.  Deployed  from  CONUS  to  overseas,  but  to  a  location  other  than  Middle  East. 
4.5 

c.  Deployed  from  overseas  to  CONUS.  1.2 

d.  Deployed  within  the  U.S.  0.8 

e.  Did  not  deploy  -  stayed  in  the  same  location  as  before.  80.1 


49.  If  you  deployed  in  support  of  the  conflict  was  this  deployment  with  your  entire 
unit,  part  of  your  unit,  or  individually? 

a.  Does  not  apply  -  did  not  deploy.  81.6 

b.  Deployed  individually,  not  with  my  unit.  3.1 
c  Deployed  with  part  of  my  unit.  12.7 

d.  Deployed  with  entire  unit.  2.6 


50.  During  the  period  of  the  conflict  (from  August  1990  to  February  1991)  which  of 
the  following  types  of  training  activities  did  you  participate  in?  (Mark  all  that 
apply) 

a.  Normal  OJT  training.  563 

b.  Special  OJT  to  learn  how  to  operate/maintain  equipment  in  the  desert 
environment.  9.1 

c  Special  unit  training  to  learn  how  to  live  in  the  desert  environment  7.3 

d.  Additional  ancillary  training  for  threats  such  as  chemical/biological  warfare. 
24.9 

e.  Training  for  working  and  living  within  an  Arabic  culture.  6.9 

f.  Normally  scheduled  ancillary  training.  45.9 

g.  Other.  17.7 
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51.  During  the  period  of  conflict,  that  part  of  your  duty  time  devoted  to  training: 

a.  Increased  substantially  7.4 

b.  Increased  slightly  18.6 
c  Did  not  change  52.6 

d.  Decreased  slightly  8.5 

e.  Decreased  substantially  12.9 


52.  How  well  trained  were  you  for  your  duties  during  Desert  Storm?  (Le.,  were  you 
trained  on  the  right  things?) 

a.  Very  well  trained  48.3 

b.  Somewhat  well  trained  29.9 

c.  Little  useful  training  10.5 

d.  No  useful  training  11.3 


Thank  you  for  completing  this  survey.  Please  return  this  survey  form  and  your 
answer  sheet  to  the  person  who  gave  them  to  you. 
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AIR  FORCE  SQUADRON  LEVEL  TRAINING  PROJECT 
INDIVIDUAL  INTERVIEW 
TRAINING  MANAGERS 


DfTRQDUCTIQK 

Good  morning  (afternoon).  Sergeant  (Airman) _ . 

I  am _ _  a  consultant  with  HAY  Systems,  Inc.,  a  Washington, 

DC-based  management  consultant  firm.  You  have  been  selected  by  the  Air  Force  to  take  pan  in  a 
survey  covering  various  aspects  of  training  in  the  Air  Force.  HAY  Systems  is  performing  this 
survey  under  contract  to  the  Technical  Training  Research  Division,  Human  Resources  Directorate 
of  the  Armstrong  Laboratory  at  Brooks  Air  Force  Base,  Texas. 

1  am  going  to  be  asking  you  a  few  questions  concerning  your  experience  with  training  in 
the  Air  Force.  The  information  you  will  be  providing  us  will  be  used  for  research  on  training  at  the 
unit  level  in  the  Air  Force.  Your  assistance  will  help  us  better  understand  the  viewpoint  of  those 
involved  in  base  level  training  and  the  on-site  training  process. 

The  information  you  provide  will  be  kept  in  strictest  confidence,  and  in  no  way  attributed  to 
you  personally.  The  background  information  that  you  will  be  asked  to  provide  will  only  be  used  to 
help  assure  that,  overall,  the  people  being  interviewed  are  representative  of  the  total  population  in 
the  Air  Force. 

Please  feel  free  to  stop  me  if  1  ask  a  question  that  is  not  clear  or  you  want  repeated.  Since 
this  same  interview  is  being  conducted  with  a  large  number  of  people,  some  of  the  questions  may 
not  apply  to  you  or  your  situation.  Do  not  hesitate  to  say  that  you  do  not  know  if  that  is  the  case. 
The  interview  should  take  less  than  an  hour  of  your  time. 

Do  you  have  any  questions? 
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1.  BACKGROUND  The  first  few  questions  concern  your  background: 

1.1  To  what  type  unit  are  you  assigned?  (e.g..  Maintenance,  Security  Police, 
Personnel,  Civil  Engineering,  Supply, 

Transportation) _ _ _ _ _ 

1.2  What  is  your  unit's  parent  Major  Command?  (e.g.,  TAC,  SAC,  MAC,  AFMPC, 

AFCA) _ 

1.3  What  is  your  current  duty  AFSC? _ _ 

(If  a  75XXX:] 

1 .3. 1  What  AFSC  did  you  hold  immediately  prior  to  crossing  into  the  75-career 

field? _ 

1.3.2  How  long  have  you  been  a  75XXX? _ 

1 .4  How  long  have  you  been  in  the  Air  Force? _ 

1.5  For  which  unit(s)  do  you  serve  as  training  manager? _ 


1.6  Is  training  management: 

1.6.1  Y our  primary  duty? _ 

1 .6.2  An  additional  duty? _ 

1 .7  For  how  long  have  you  served  in  this  capacity? _ 

1.8  Were  you  serving  in  a  Training  Manager  position  prior  to  the  18  July  1990  change 
to  AFR  50-23  (Enlisted  Specialty  Training)? 

1.8.1  No _  (Go  to  next  section.) 

1.8.2  Yes _ 
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1 .9  Has  your  role  as  Training  Manager  changed  significantly  since  publication  of  the  18 
July  1990  AFR  50-23? 


1.9.1  No 

(Go  to  next  section. 1 

1.9.2  Yes 

.  In  the  following  v,ay(s): 

1.9.2. 1 

I  have  less  control  over  the  unit's  training  program 

1.9. 2. 2 

I  no  longer  have  the  regulatory  "hammer"  to  enforce  training 
procedures 

1.9. 2.3 

My  role  as  a  Training  Manager  has  been  significantly 
diminished 

1.9. 2. 4 

My  role  as  a  Training  Manager  has  been  significantly 
enhanced  or  expanded 

1.9. 2.5 

I  have  more  flexibility  to  tailor  the  unit  s  training  program  to 
local  needs _ 

1. 9.2.6 

Other  (specifvl 

2.  TRAINING  NEEDS 

2. 1  Can  you  predict  when  there  will  be  greater  or  lesser  training  needs  in  your  unit? 

2.1.1  No _  (Go  to  question  2.41 

2.1.2  Yes _ 

2.2  How  far  ahead  can  you  predict? _ 

2.3  What  helps  you  to  predict? _ _ _ 

2.4  During  the  past  year,  have  your  unit  level  training  requirements: 

2.4. 1  Increased _ 

2.4.2  Decreased _ 

2.4.3  Remained  about  the  same _ (Go  to  question  2.61 
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2.5  What  has  driven  the  change  in  training  requirements? 

2.5. 1  Influx  of  entry-levei  personnel  (new  enlistees  or  cross-trainees). 

2.5.2  New  equipment _ 

2.5.3  Increased  manning _ 

2.5.4  Decreased  manning _ 

3.5.5  Decreased  number  of  Trainers _ 

3.5.6  Increased  number  of  Trainers _ 

2.5.7  Increased  mission  requirements _ 

2.5.8  Decreased  mission  requirements _ 

2.5.9  Revised  training  standards 

2.5. 10  Change  in  command  emphasis  on  training _ 

2.5. 1 1  AFSC  mergers/consoiidations _ 

2.5.12  Other  (specify) _ 


191 


2.6  Have  there  been  any  AFSC  mergers  or  skill  consolidations  within  your  unit? 

2.6.1  Yes _ 

2.6. 1 . 1  How  have  these  mergers  or  consolidations  affected  unit  level 
training  requirements?. _ _ _ 


2.6.2  No _ 

2.6.2. 1  If  AFSC  mergers  or  consolidations  were  directed  within 

your  unit,  what  changes  do  you  expect  would  result  in  the 
unit’s  training  requirements? _ 


2.7  Has  there  been  an  introduction  of  new  technologies  or  equipment  in  your  unit 
which  required  an  increase  in  unit  level  skill  or  task  qualification  training? 

2.7.1  No _  (Go  to  next  section.) 

2.7.2  Yes _ 

2.7.2. 1  What  training  methods  or  technologies  were  required  to 

support  this  unit  level  skill  or  qualification 
training? _ 
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3. 


3. 1  Are  you  involved  in  scheduling  training  activities? 

3.1.1  No _  (Go  to  next  section.) 

3.1.2  Yes _ 

3.2  How  is  this  scheduling  accomplished? 

3.2.1  Manually  (Go  to  question  3.3) 

3.2.2  Automated  scheduling  system _ 

3.2.2. 1  What  is  the  name  of  the  automated  scheduling  system 

used? _ 

3. 2.2. 2  Where  was  the  system  developed? 

3.2.2.2.1  Locally _ 

3. 2.2. 2. 2  Elsewhere  (specify  where) _ 

3. 2. 2.3  What  are  its  good  points  or  advantages? 

3. 2.2.3. 1  Speed 

3. 2.2.3. 2  Accuracy _ 

3.2.2.3.3  Easily  changed _ 

3. 2. 2. 3. 4  More  efficient  than  manual  system _ 

3. 2.2. 3. 5  Other  (specify) _ 

3. 2. 2. 3. 6  None _ 

3. 2. 2. 4  What  are  its  drawbacks  or  disadvantages? 

3. 2. 2. 4.1  Too  complex _ 

3. 2. 2. 4.2  Too  cumbersome _ 

3. 2.2. 4. 3  Too  costly _ 

3. 2. 2. 4. 4  Too  inflexible _ 

3. 2.2.4. 5  Other  (specify) _ 

3.2.2. 4.6  None _ 
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3.3  What  are  the  most  significant  problems  you  have  with  scheduling? 

3.3.1  Unit  mission  demands _ 

3.3.2  Availability  of  Trainers  or  Trainees _ 

3.3.3  Last-minute  changes _ 

3.3.4  Lack  of  communication  from  supervisors  or  trainers _ 

3.3.5  Lack  of  an  automated  scheduling  system _ 

3.3.6  Other  (specify) _ _ 

3.3.7  None _ 
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4.  TRAINING  METHQDS/TF.(~!HNOLOGIES 

4.1  In  a  training  program,  a  number  of  methods  and  tools  can  be  used,  such  as 
overhead  projectors,  sound-on-slide,  videos,  computers,  etc.  1  am  going  to  read 
you  a  list  of  some  of  these.  Please  tell  me  which  ones  are  used  for  training  in  your 
unit. 

4.1.1  Computer  based  training  (CBT) _ 

4. 1 .2  Interactive  video  disks  (IVD) _ 

4.1.3  Videotapes _ 

4. 1 .4  Formal  instructor-conducted  classroom  instruction - 

4. 1 .5  Self-study  books  or  other  printed  materials _ 

4.1.6  Simulators  or  mocK-ups _ 

4. 1 .7  Operatioi.il  mission  equipment _ 

4.1.8  Non-operational  mission  equipment _ 

4.2  What  are  the  advantages  of  the  methods  used  in  your  unit? 

4.2. 1  Saves  money _ 

4.2.2  Saves  rime _ 

4.2.3  Requires  less  Instructor  time _ 

4.2.4  Provides  real-time  feedback _ 

4.2.5  Allows  for  hands-on  training _ 

4.2.6  Less  disruptive  to  operational  activities _ 

4.2.7  Provides  realism _ 

4.2.8  Other  (specify) _  _ _ _ _ _ — 

4.2.9  None _ 
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4.3  What  arc  the  disadvantages  of  the  methods  used  in  your  unit? 


4.3.1  Costs  too  much  money _ 

4.3.2  Costs  or  wastes  time _ 

4.3.3  Requires  more  Instructor  time _ 

4.3.4  Does  not  provide  real-time  feedback _ 

4.3.5  Does  not  allow  for  hands-on  training _ 

4.3.6  Too  disruptive  to  operational  activities _ 

4.3.7  Does  not  provide  realism _ 

4.3.8  Other  (specify) _ 

4.3.9  None _ 

4.4  Have  any  training  methods  been  used  in  the  past  that  are  no  longer  used? 

4.4.1  No _  (Go  to  next  section.) 

4.4.2  Yes _ 

4.5  Why  were  these  methods  dropped? 

4.5.1  Replaced  by  better  methods _ 

4.5.2  Too  expensive _ 

4.5.3  Wasted  too  much  time _ 

4.5.4  Designed  for  equipment  which  became  obsolete,  or  left  the  inventory. 

4.5.5  Other  (specify) _ 
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5.  TRAINING  DEVELOPMENT 


5. 1  Are  any  aspects  of  your  unit  training  developed  locally  (mock-ups.  training 
equipment,  exercises,  courses  and/or  course  materials)? 

5.1.1  No _  (Go  to  Question  5.3) 

5.1.2  Yes _ 

5.2  About  how  much  training  development  is  done  locally  versus  being  developed  by 
outside  sources? 

5.2. 1  Over  50  percent _ 

5.2.2  25-50  percent _ 

5.2.3  10-25  percent _ 

5.2.4  Less  than  10  percent _ 

5.3  Do  you  now,  or  have  you  ever,  participated  in  training  development? 

5.3.1  No _  (Go  to  next  section.) 

5.3.2  Yes _ 

5.4  What  technologies  have  you  used  to  develop  training? 

5.4. 1  Instructional  System  Development  (ISD) _ 

5.4.2  Automated  system  (specify) _ 

5.4.3  Other  (specify) _ 
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5.5  If  ISD  was  not  used,  why  not? 

5.5.1  Not  applicable.  Used  ISD_ _ 

5.5.2  Not  familiar  with  it _ 

5.5.3  Too  complex _ 

5.5.4  Too  cumbersome _ 

5.5.5  Not  applicable  to  the  type  training  development  I  was  doing_ 

5.5.6  Too  time-consuming _ 

5.5.7  Too  costly _ 

5.5.8  Other  (specify) _ _ 

5.6  What  guidance  do  you  receive  or  use  for  the  development  of  training 

content? _ 

5.6.1  What  or  who  is  the  source  of  this 

guidance? _ _ 
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6.  DOCUMENTATION 


6.1  Do  you  document  training  accomplished? 

6.1.1  No _  (Go  to  next  section.! 

6.1.2  Yes _ It  is  done: 

6. 1.2.1  Manually _ 

6. 1 .2.2  Using  an  automated  system _ 

6. 1 .2.2. 1  System  was  locally  developed _ 

6. 1 .2.2.2  System  was  developed  elsewhere  (specify 

where) _ 

6.2  What  arc  the  most  significant  problems  you  have  with  documentation? 

6.2. 1  It  is  paperwork-intensive _ 

6.2.2  It  takes  too  much  time _ 

6.2.3  It  is  too  cumbersome _ 

6.2.4  It  isn’t  standardized  across  bases _ 

6.2.5  I  question  its  value  as  currently  structured _ 

6.2.6  It  should  be  automated _ 

6.2.7  Other  (specify) _ 

6.2.8  None.  1  have  no  problems  with  it _ 
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7.  TRAINING  EVALUATION 


7.1  Do  you  personally  evaluate  the  results  of  training? 

7.1.1  No _ 

7.1.2  Yes _ 

7.2  What  methods  are  used  for  training  evaluation  in  your  unit? 

7.2.1  Observe  trainee  performance  (over-the-shoulder) _ 

7.2.2  Verbally  question  trainees  about  job  performance _ 

7.2.3  Interview  supervisors/trainers  of  trainees _ 

7.2.4  Written  examinations _ 

7.2.5  Other  (specify) _ 

7.2.6  Do  not  know _ 

7.3  Are  any  automated  «”stems  used  or  available  for  performing  evaluations? 

7.3.1  No _  (Go  to  next  section. ) 

7.3.2  Yes _ 

7.3.2. 1  What  are  they? _ 

7. 3. 2. 2  Were  they: 

7. 3.2.2. 1  Locally  developed? _ 

7. 3. 2. 2. 2  Developed  elsewhere?  (specify) _ 
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8. 


8. 1  Do  you  have  contacts  with  other  training  managers  at  this  base? 

8.1.1  No _ 

8. 1.1.1  Why  not?. _ _ _ 

(Go  to  next  section.! 

8.1.2  Yes _ 

8.2  What  types  of  contact  do  you  have? 

8.2. 1  Informal,  as-required _ 

8.2.2  Formal,  as-required _ 

8.2.3  Formal,  on  a  regularly  scheduled  basis _ 

8.2.3. 1  How  frequently? _ 

8 . 3  Are  these  contacts  useful  to  you  as  a  training  manager? 

8.3.1  No _ 

8.3.1. 1  Why  not? _ _ _ _ _ 

8.3.2  Yes _ 

8.3.2. 1  In  what  way? 

8.3.2. 1 . 1  We  share  information  about  various  training 

technologies _ 

8. 3. 2. 1.2  We  leam  from  one  another _ 

8.3.2. 1 .3  We  are  able  to  solve  common  problems _ 

8.3.2. 1 .4  We  coordinate  such  things  as  scheduling  of  shared 

facilities _ 

8. 3. 2. 1.5  Other  (specify) _ 
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9. 1  Do  you  believe  that  increased  levels  of  training  can  be  accomplished  by  your  unit 
without  having  a  negative  impact  on  operational  capability? 

9.1.1  Yes _ 

9. 1 . 1 . 1  By  how  much?  (Percentage  increase) _ 

9.1.2  No _ 

9.2  If  the  Air  Force  reduced  ATC  resident  training,  and  shifted  this  training  to  the  unit 
level,  what  do  you  believe  would  be  the  impact? 

9.2. 1  Unit  mission  would  suffer  •  readiness  would  be  degraded _ 

9.2.2  We  would  need  more  manpower _ 

9.2.3  Quality  of  training  would  suffer 

9.2.4  We  would  need  to  find  new  ways  of  doing  business _ 

9.2.5  We  would  need  to  find  new  ways  to  perform  the  training  mission _ 

9.2.6  Quality  of  training  would  improve _ 

9.2.7  Unit  would  benefit  because  we  could  tailor  the  training  program  to  our 

specific  needs _ 

9.2.8  It  would  cause  us  to  seek  state-of-the-art  training  technologies  such  as 

enhanced  methods  and  tools _ 

9.2.9  Other  (specify) _ 
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10. 1  In  your  opinion,  what  works  well  in  your  unit  training  program  or 
activities? _ 


10.2  What  problems  are  there?. 


10.3  Are  there  any  training  technologies  that  are  not  available  to  you  that  would  benefit 
your  training  program  or  activities? 

10.3.1  I  don’t  know  of  any _ 

10.3.2  Yes _ 

10.3.2.1  What  are  they? _ 


10.4  Do  you  have  any  suggestions  for  improving  unit  level  training*) 
10.4.1  No _ 

1L.4.2  Yes _ _ _ 


11.  PARTICIPATION  IN  DESERT  SHIELD/DESERT  STORM 

11.1  Did  you  participate  in  Desert  Shield  and/or  Desert  Storm  (conflict  with  Iraq,  August 
1990  to  February  1991)? 

11.1.1  Yes _  (Go  to  DS  Interview  Form.) 

11.12  No _  THANK  THE  RESPONDENT 


203 


Appendix  C-2 
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AIR  FORCE  SQUADRON  LEVEL  TRAINING  PROJECT 
INDIVIDUAL  INTERVIEW 
TRAINERS 


INTRODUCTION 

Good  tnoming  (afternoon).  Sergeant  (Airman) _ . 

I  am _ _  a  consultant  with  HAY  Systems,  Inc.,  a  Washington, 

DC-based  management  consultant  firm.  You  have  been  selected  by  the  Air  Force  to  take  pan  in  a 
survey  covering  various  aspects  of  training  in  the  Air  Force.  HAY  Systems  is  performing  this 
survey  under  contract  to  the  Technical  Training  Research  Division,  Human  Resources  Directorate 
of  the  Armstrong  Laboratory  at  Brooks  Air  Force  Base,  Texas. 

I  am  going  to  be  asking  you  a  few  questions  concerning  your  experience  with  training  in 
the  Air  Force.  The  information  you  will  be  providing  us  will  be  used  for  research  on  training  at  the 
unit  level  in  the  Air  Force.  Your  assistance  will  help  us  better  understand  the  viewpoint  of  those 
involved  in  base  level  training  and  the  on-site  training  process. 

The  information  you  provide  will  be  kept  in  strictest  confidence,  and  in  no  way  attributed  to 
you  personally.  The  background  information  that  you  will  be  asked  to  provide  will  only  be  used  to 
help  assure  that,  overall,  the  people  being  interviewed  are  representative  of  the  total  population  in 
the  Air  Force. 

Please  feel  free  to  stop  me  if  I  ask  a  question  that  is  not  clear  or  you  want  repeated.  Since 
this  same  interview  is  being  conducted  with  a  large  number  of  people,  some  of  the  questions  may 
not  apply  to  you  or  your  situation.  Do  not  hesitate  to  say  that  you  do  not  know  if  that  is  the  case. 
The  interview  should  take  less  than  an  hour  of  your  time. 

Do  you  have  any  questions? 
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I.  BACKGROUND  The  first  few  questions  concern  your  background: 

1.1  To  what  type  unit  are  you  assigned?  (e.g..  Maintenance,  Security  Police, 
Personnel,  Civil  Engineering,  Supply, 

Transportation) _ _ _ 

1.2  What  is  your  unit's  parent  Major  Command?  (e.g.,  TAC,  SAC,  MAC,  AFMPC, 

AFCA) _ 

1.3  What  is  you.'  current  duty  AFSC? _ _ 

(If  a  75XXX:] 

1.3.1  What  AFSC  did  you  hold  immediately  prior  to  crossing  into  the  75- 

career  field? _ 

1.3.2  How  long  have  you  been  a  75XXX? _ 

1 .4  How  long  have  you  been  in  the  Air  Force? _ _ 

1.5  For  which  unit(s)  do  you  serve  as  trainer? 


1.6  Is  training: 

1.6.1  Your  primary  duty? _ 

1.6.2  An  additional  duty? _ 

1.7  For  how  long  have  you  served  in  this  capacity? _ 

1 . 8  For  what  type  of  training  are  you  currently  a  trainer? 

1.8.1  On-the-Job  Training  (OJT) _ 

1.8.2  Upgrade  Training  (UGT) _ 

1.8.3  Ancillary  Training _ 

1.8.4  Professional  Military  Education  (PME) _ 

1 .8.5  Field  Training  Detachment  (FTD) _ 

1.8.6  Other  (specify) _ 

1.9  During  the  course  of  your  career  to  this  point  in  time,  how  often  have  you  been 

assigned  trainer  duties? _ 
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1.10  How  were  you  selected  as  a  Trainer  this  time? 

1.10.1  Rank _ . 

1.10.2  Demonstrated  job-related  skills _ 

1.10.3  Demonstrated  teaching  ability _ 

1 . 10.4  Past  experience  as  a  Trainer _ 

1.10.5  Had  completed  Trainer  training _ 

1.10.6  No  one  else  available _ 

1.10.7  I  volunteered _ 

1.10.8  Other  (specify) _ 

1.11  How  were  you  selected  as  a  Trainer  in  the  past? 

1.11.1  Rank _ 

1.11.2  Demonstrated  job-related  skills _ 

1.11.3  Demonstrated  teaching  ability _ 

1.11.4  Past  experience  as  a  Trainer _ 

1.11.5  Had  completed  Trainer  training _ 

1.11.6  No  one  else  available _ 

1 . 1 1 .7  I  volunteered _ 

1.11.8  Other  (specify) _ _ 

1 . 1  i  .9  Not  applicable.  I  have  never  been  a  Trainer  in  the  past 

1.12  Are  you  Instructor-qualified? 

1.12.1  Yes _ 

1.12.2  No _ 
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1.13 


What  type  of  training  or  preparation  have  you  had  to  perform  as  a  Trainer'1 

1.13.1  Academic  Instructor  Course  (AIC) _ 

1.13.2  OJT  Supervisor  Course _ 

1.13.3  Other  formal  Trainer/Instructor  Course  (specify) _ 

1.13.4  OJT  with  another  Trainer _ 

1.13.5  Self-study _ 

1.13.6  Other  (specify) _ 

1.13.7  None _ 

1.14  Who  evaluates  your  performance  as  a  trainer? 

1.14.1  Immediate  supervisor _ 

1.14.2  Unit  commander _ 

1.14.3  Headquaiters  Squadron  Section  Commander _ 

1.14.4  First  Sergeant _ 

1 . 14.5  Unit  Training  Manager _ _ 

1.14.6  Other  (specify) _ 

1.15  On  what  basis  is  your  performance  as  a  Trainer  evaluated? 

1.15.1  Quaiifications/success  of  Tramee(s) _ 

1.15.2  Quality  of  the  product _ 

1.15.3  My  demonstrated  training  abilities _ 

1 .15.4  My  paperwork/documentation  is  up-to-date _ 

1.15.5  Other  (specify) _ 

1.15.6  Not  evaluated  as  a  Trainer,  or  Don't  know _ 

1.16  What  portion  of  your  normal  duty  week  do  you  estimate  you  spend  on  unit  level 

training? _ (%) 

1.17  Were  you  serving  in  a  Trainer  position  prior  to  the  1 8  July  1 990  change  to  AFR  50- 
23  (Enlisted  Specialty  Training)? 
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1.17.1 


No. 


(Go  to  next  section. 1 


1.17.2  Yes _ 

1.18  Has  your  role  as  a  Trainer  changed  significantly  since  publication  of  the  18  July 

1990  AFR  50-23? 

1.18.1  No _  (Go  to  next  section. 1 

1 . 1 8.2  Yes _  In  the  following  way(s): 

1.18.2.1  I  have  less  control  over  my  training  program _ 

1 . 1 8.2.2  1  no  longer  have  the  regulatory  "hammer"  to  enforce  training 

procedures _ 

1 . 1  S.2.3  My  role  as  a  Trainer  has  been  significantly  diminished _ 

1 . 1 8.2.4  My  role  as  a  Trainer  has  been  significantly  enhanced  or 

expanded _ 

1.18.2.5  1  have  more  flexibility  to  tailor  my  training  program  to  local 

needs _ 

1.18.2.6  Other  (specify) _ 
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2.  PLANNING 


2. 1  Arc  you  involved  in  planning  training  for  your  unit? 

2.1.1  No _  (Go  to  next  section.) 

2.1.2  Yes _ 

2.2  What  would  you  describe  as  realistic  long-range  planning  for  your  unit's  program0 

2.2. 1  Six  months  or  less _ 

2.2.2  One  year _ 

2.2.3  Eighteen  months _ 

2.2.4  Two  years  or  more _ 

2.3  Do  you  have  a  formal  training  process  or  Training  Plan? 

2.3.1  No _  (Go  to  next  section.) 

2.3.2  Yes _ 

2.3.2. 1  Is  your  planning  process  automated,  or  is  it  done  manually? 

2. 3. 2. 1.1  Automated _ 

2.3.2. 1.2  Manual _ 

2. 3. 2. 2  What  factors  are  considered  in  planning  training? 
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2. 3. 2. 3  How  often  is  your  plan  updated? 

2. 3. 2. 3.1  Continuously _ 

2.3.2. 3. 2  Monthly _ 

2. 3. 2. 3. 3  Quarterly _ 

2. 3. 2. 3. 4  Biannually _ 

2. 3. 2. 3. 5  Annually _ 

2. 3. 2. 3. 6  Other  (Specify) _ 

2. 3. 2. 4  Once  your  plan  is  established,  does  it  usually  require 
significant  update  or  change? 

2. 3. 2. 4.1  No _ 

2. 3. 2. 4. 2  Yes _ 

2. 3.2.5  What  types  of  adjustments,  if  any,  are  usually  required? 
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3.  TRAINING  NEEDS 


3. 1  Can  you  predict  when  there  will  be  greater  or  lesser  training  needs? 

311  No _  (Go  to  question  3.4) 

3.1.2  Yes _ 

3.2  How  far  ahead  can  you  predict? _ 

3.3  What  helps  you  to  predict? _ 

3.4  During  the  past  year,  have  your  training  requirements: 


3.4.1 

Increased 

3.4.2 

Decreased 

3.4.3 

Remained  about  the  same  (Go  to  Question  3.6) 

What  has  driven  the  change  in  training  requirements? 

3.5.1 

Influx  of  entry-level  personnel  (new  enlistees  or  cross¬ 
trainees) 

3.5.2 

New  equipment 

3.5.3 

Increased  manning 

3.5.4 

Decreased  manning 

3.5.5 

Decreased  number  of  Trainers 

3.5.6 

Increased  number  of  Trainers 

3.5.7 

Increased  mission  requirements 

3.5.8 

Decreased  mission  requirements 

3.5.9 

Revised  training  standards 

3.5.10 

Change  in  command  emphasis  on  training 

3.5.11 

AFSC  mergers/consolidations 

3.5.12 

Other  (specify) 
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3.6  Have  there  been  any  AFSC  mergers  or  skill  consolidations  within  your  unit0 


3.6.1  Yes. 


3.6. 1 .  i  How  have  these  mergers  or  consoiidanons  affected  your 
training  requirements? 


3.6.2  No. 


3.6.2. 1  If  AFSC  mergers  or  consolidations  were  directed  within 

your  unit,  what  changes  do  you  expect  would  result  in  your 
training  requirements? _ 


3.7  Has  there  been  an  introduction  of  new  technologies  or  equipment  in  your  unit 
which  required  an  increase  in  unit  level  skill  or  task  qualification  training? 

3.7.1  No _  (Go  to  next  section.! 

3.7.2  Yes _ 

3.7.2. 1  What  training  methods  or  technologies  were  required  to 
support  this  unit  level  skill  or  qualification 
training? _ 
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4.  SCHEDULING 

4. 1  Are  you  involved  in  scheduling  training  activities? 

4.1.1  No _  {Go  to  next  section.! 

4.1.2  Yes _ 

4.2  How  is  this  scheduling  accomplished? 

4.2.1  Manually  IGo  to  question  4.31 

4.2.2  Automated  scheduling  system _ 

4.2.2. 1  Where  was  the  system  developed? 

4.2.2. 1.1  Locally _ 

4.2.2. 1 .2  Elsewhere  (specify  where) _ 

4.2.2.2  What  are  its  good  points  or  advantages? 

4.2.2.2.1  Speed _ 

4. 2.2. 2. 2  Accuracy _ 

4. 2.2. 2.3  Easily  changed _ 

4.2.2.2.4  More  efficient  than  manual  system _ 

4. 2. 2. 2. 5  Other  (specify) _ 

4. 2. 2. 2. 6  None _ 

4. 2.2. 3  What  are  its  drawbacks  or  disadvantages? 

4. 2. 2. 3.1  Too  complex _ 

4. 2. 2. 3. 2  Too  cumbersome _ 

4. 2. 2. 3. 3  Too  costly _ 

4.2.2. 3.4  Too  inflexible _ 

4. 2.2. 3. 5  Other  (specify) _ 

4. 2.2. 3.6  None _ 


217 


What  are  the  most  significant  problems  you  have  with  scheduling0 

4.3.1  U  nit  mission  demands _ 

4.3.2  Availability  of  Trainers  or  Trainees _ 

4.3.3  Last-minute  changes _ 

4.3.4  Lack  of  communication  from  supervisors _ 

4.3.5  Lack  of  an  automated  scheduling  system _ 

Other  (specify) _ 


4.3.6 


5.  TRAINING  METHODSn-ECHNQLOGfES 


5.1  In  a  training  program,  a  number  of  methods  and  tools  can  be  used,  such  as 
overhead  projectors,  sound-on-slide,  videos,  computers,  etc.  I  am  going  to  read 
you  a  list  of  some  of  these.  Please  tell  me  which  ones  are  used  for  training  in  your 
unit. 

5.1.1  Computer  based  trai  n  ing  ( CBT) _ 

5. 1 .2  Interactive  video  disks  (IVD) _ 

5.1.3  V  ideotapes _ 

5. 1 .4  Formal  instructor-conducted  classroom  instruction _ 

5.1.5  Self-study  books  or  other  printed  materials _ 

5.1.6  Simulators  or  mock-ups _ 

5. 1 .7  Operational  mission  equipment _ 

5.1.8  Non-operational  mission  equipment _ 

5.2  What  are  the  advantages  of  the  methods  used  in  your  unit? 

5.2.1  Saves  money _ 

5.2.2  Saves  time _ 

5.2.3  Requires  less  instructor  time _ 

5.2.4  Provides  real-time  feedback _ 

5.2.5  Allows  for  hands-on  training _ 

5.2.6  Less  disruptive  to  operational  activities _ 

5.2.7  Provides  realism _ 

5.2.8  Other  (specify) _ 

5.2.9  None _ 
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5.3  What  are  the  disadvantages  of  the  methods  used  in  your  unit? 

5.3.1  Costs  too  much  money _ 

5.3.2  Wastes  time _ 

5.3.3  Requires  more  Instructor  time _ 

5.3.4  Does  not  provide  real-time  feedback _ 

5.3.5  Does  not  allow  for  hands-on  training _ 

5.3.6  Too  disruptive  to  operational  activities _ 

5.3.7  Does  not  provide  realism _ 

5.3.8  Other  (specify) _ _ 

5.3.9  None _ 

5.4  Have  any  training  methods  been  used  in  the  past  that  are  no  longer  used? 

5.4.1  No _  (Go  to  next  section.) 

5.4.2  Yes _ 

5.5  Why  were  these  methods  dropped? 

5.5. 1  Replaced  by  better  methods _ 

5.5.2  Too  expensive _ 

5.5.3  Wasted  too  much  time _ 

5.5.4  Designed  for  equipment  which  became  obsolete,  or  left  the 

inventory _ 

5.5.5  Other  (specify) _ 
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6.  TRAINING  DEVELOPMENT 

6. 1  Are  any  aspects  of  your  unit  training  developed  locally  (mock-ups.  training 
equipment,  exercises,  courses  and/or  course  materials)? 

6.1.1  No _  (Go  to  question  6.3) 

6.1.2  Yes _ 

6.2  About  how  much  training  development  is  done  locally  versus  being  developed  by 
outside  sources? 

6.2. 1  Over  50  percent _ 

6.2.2  25-50  percent _ 

6.2.3  10-25  percent _ 

6.2.4  Less  than  10  percent _ 

6.3  Do  you  now,  or  have  you  ever,  participated  in  training  development? 

6.3.1  No _  (Go  to  next  section.) 

6.3.2  Yes _ 

6.4  What  technologies  have  you  used  to  develop  training? 

6.4. 1  Instructional  System  Development  (ISD) _ 

6.4.2  Automated  system  (specify) _ 

6.4.3  Other  (specify) _ 
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6.5  If  ISD  was  not  used,  why  noi? 

6.5.1  Not  applicable.  Used  ISD _ 

6.5.2  Not  familiar  with  it _ 

6.5.3  Too  complex _ 

6.5.4  Too  cumbersome _ 

6.5.5  Not  applicable  to  the  type  training  development  I  was  doin 

6.5.6  Too  time-consuming _ 

6.5.7  Too  costly _ 

6.5.8  Other  (specify) _ 

6.6  What  guidance  do  you  receive  or  use  for  the  development  of  training 

content? _ 


6.6. 1  What  or  who  is  the  source  of  this 

guidance? _ 
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7.1  Do  you  document  training  accomplished? 

7.1.1  No _  (Go  to  next  section.) 

7.1.2  Yes _ 

7.2  How  is  your  training  documentation  accomplished? 

7.2.1  Manually _ 

7.2.2  Using  an  automated  system _ 

7.2.2. 1  What  is  the  name  of  the  automated  system? 


7. 2.2. 2  Where  was  the  system  developed? 

7. 2. 2. 2.1  Locally _ 

7. 2.2. 2. 2  Elsewhere  (specify  where) _ 

7.3  What  are  the  most  significant  problems  you  have  with  documentation? 

7.3. 1  It  is  paperwork-intensive _ 

7.3.2  It  takes  too  much  time _ 

7.3.3  It  is  too  cumbersome _ 

7.3.4  It  isn't  standardized  across  bases _ 

7.3.5  I  question  its  value  as  currently  structured _ 

7.3.6  It  should  be  automated _ 

7.3.7  Other  (specify) _ 

7.3.8  None.  I  have  no  problems  with  it _ 
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TRAINING  EVALUATION 

8.1  Do  you  personally  evaluate  the  results  of  training? 

8.1.1  No _  (Go  to  next  section.) 

8.1.2  Yes _ 

8.2  What  methods  do  you  use  for  training  evaluation? 

8.2. 1  Observe  trainee  performance  (over- the- shoulder) _ 

8.2.2  Verbally  question  trainees  about  job  performance _ 

8.2.3  Interview  supervisors  of  trainees _ 

8.2.4  Written  examinations _ 

8.2.5  Other  (specify) _ 

8.3  Are  any  automated  systems  used  or  available  for  performing  evaluations? 

8.3.1  No _  (Go  to  next  section.) 

8.3.2  Yes _ 

8.3.2. 1  What  are  they? _ 

8. 3.2. 2  Were  they: 

8 . 3 . 2 . 2 . 1  Local  ly  developed? _ 

8. 3. 2. 2. 2  Developed  elsewhere?  (specify) _ 
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°  1  About  how  often  do  you  have  contact  with  your  unit  Training 
Man  ager? _ 

9.2  What  types  of  contact  do  you  have? 

9.2.1  Informal,  as-required _ 

9.2.2  Formal,  as-required _ 

9.2.3  Formal,  on  a  regularly  scheduled  basis _ 

9.2.3. 1  How  frequently? _ 

9.3  Arc  these  contacts  useful  to  you  as  a  trainer? 

9.3.1  No _ 

9.3.1. 1  Why  not? _ 

9.3.2  Yes _ 

9  3.2.3  In  what  way? 


9.3.2.1.1 

We  share  information  about  various  training 
technologies 

9.3.2.1.2 

We  learn  from  one  another 

9.3.2. 1.3 

We  are  able  to  solve  common  problems 

9.3.2.1.4 

We  coordinate  such  things  as  scheduling  of  shared 
facilities 

9. 3.2. 1.5 

He/she  helps  me  with  "problem  trainees" 

9.3.2.J.6 

Other  (specifvl 
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10.  IMPACT  OF  EXPANDED  SQUADRON  LEVEL  TRAINING 


10. 1  Do  you  believe  that  increased  levels  of  training  can  be  accomplished  by  your  unit 
without  having  a  negative  impact  on  operational  capability? 

10.1.1  Yes _ 

10. 1 . 1 . 1  By  how  much?  (Percentage  increase) _ 

10.1.2  No _ 

10.2  If  the  Air  Force  reduced  ATC  resident  training,  and  shifted  this  training  to  the  unit 
level,  what  do  you  believe  would  be  the  impact? 

10.2.1  Unit  mission  would  suffer  -  readiness  would  be  degraded _ 

10.2.2  We  would  need  more  manpower _ 

10.2.3  Quality  of  training  would  suffer 

10.2.4  We  would  need  to  find  new  ways  of  doing  business _ 

10.2.5  We  would  need  to  find  new  ways  to  perform  the  training  mission _ 

10.2.6  Quality  of  training  would  improve _ 

10.2.7  Unit  would  benefit  because  we  could  tailor  the  training  program  to  our 

specific  needs _ 

10.2.8  It  would  cause  us  to  seek  state-of-the-art  training  technologies  such  as 

enhanced  methods  and  tools _ 

10.2.9  Other  (specify) _ 
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11.  OPINIONS  ABOUT  TRAINING 

11.1  In  your  opinion,  what  works  well  in  your  unit  training  program  or 
activities? _ 


11.2  What  problems  are  there?. 


1 1 .3  Are  there  any  training  technologies  that  are  not  available  to  you  that  would  benefit 
your  training  program  or  activities? 

1 1.3.1  I  don't  know  of  any _ 

11.3.2  Yes _ 

1 1 .3.2. 1  What  are  they? _ 


1 1 .4  Do  you  have  any  suggestions  for  improving  unit  level  training? 

11.4.1  No _ 

1 1.4.2  Yes _ 


12.  PARTICIPATION  INDESERT  SHIELD/DESERT  STORM 

1 2. 1  Did  you  participate  in  Desen  Shield  and/or  Desen  Storm  (conflict  with  Iraq,  August 
1990  to  February  1991)? 


12.1.1 

Yes _ 

_  (Go  to  DS  Interview  Form.) 

12.1.2 

No _ 

_  THANK  THE  RESPONDENT 
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Appendix  C-3 
Trainee 

Interview  Guide 
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AIR  FORCE  SQUADRON  LEVEL  TRAINING  PROJECT 
INDIVIDUAL  INTERVIEW 
TRAINEES 


INTRODUCTION 

Good  morning  (afternoon),  Sergeant  (Airman) _ . 

I  am _ _  a  consultant  with  HAY  Systems,  Inc.,  a  Washington, 

DC-based  management  consultant  firm.  You  have  been  selected  by  the  Air  Force  to  take  pan  in  a 
survey  covering  various  aspects  of  training  in  the  Air  Force.  HAY  Systems  is  performing  this 
survey  under  contract  to  the  Technical  Training  Research  Division,  Human  Resources  Directorate 
of  the  Armstrong  Laboratory  at  Brooks  Air  Force  Base,  Texas. 

I  am  going  to  be  asking  you  a  few  questions  concerning  your  experience  with  training  in 
the  Air  Force.  The  information  you  will  be  providing  us  will  be  used  for  research  on  training  at  the 
unit  level  in  the  Air  Force.  Your  assistance  will  help  us  better  understand  the  viewpoint  of  those 
involved  in  base  level  training  and  the  on-site  training  process. 

The  information  you  provide  will  be  kept  in  strictest  confidence,  and  in  no  way  attributed  to 
you  personally.  The  background  information  that  you  will  be  asked  to  provide  will  only  be  used  to 
help  assure  that,  overall,  the  people  being  interviewed  are  representative  of  the  total  population  in 
the  Air  Force. 

Please  feel  free  to  stop  me  if  1  ask  a  question  that  is  not  clear  or  you  want  repeated.  Since 
this  same  interview  is  being  conducted  with  a  large  number  of  people,  some  of  the  questions  may 
not  apply  to  you  or  your  situation.  Do  not  hesitate  to  say  that  you  do  not  know  if  that  is  the  case. 
The  interview  should  take  less  than  an  hour  of  your  time. 

Do  you  have  any  questions? 
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BACKGROUND  The  first  few  questions  concern  your  background: 

1.1  To  what  type  unit  are  you  assigned?  (e.g.,  Maintenance,  Security  Police. 
Personnel,  Civil  Engineering,  Supply, 

Transportation ) _ 


1.2 

1.3 

1.4 

1.5 


1.6 


What  is  your  unit’s  parent  Major  Command?  (e.g.,  TAC,  SAC,  MAC,  AFMPC, 
AFCA) _ 

What  is  your  duty  AFSC? _ 


How  long  have  you  been  in  the  Air  Force?. 


Are  you  currently  taking  pan  in  any  kind  of  training? 


1.5.1 


Yes. 


It  is  of  the  following  type: 


1.5.2 


1.5. 1.1 

On-the-Job  (OJT)  Upgrade  Training _ 

1.5. 1.2 

Field  Training  Detachment  (FID) 

1.5. 1.3 

Studying  CDCs 

1.5. 1.4 

Ancillary  Training 

1.5. 1.5 

Professional  Military  Education  (PME) 

1.5. 1.6 

Other  (specify) 

No _ 

_  What  was  the  most  recent  training  that  you  c 

1.5.2. 1 

On-the-Job  (OJT)  Upgrade  Training 

1.5.2. 2 

Field  Training  Detachment  (FID) 

1.5. 2. 3 

Studying  CDCs 

1.5. 2.4 

Ancillary  Training 

1.5. 2. 5 

Professional  Military  Education  (PME) 

1.5. 2. 6 

Other  (specify) 

1.5. 2.7 

None 

What  ponion  of  your  normal  duty  week  is/was  spent  on  unit  level  training? 

<%) 
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2. 


2. 1  How  would  you  compare  the  training  you  receive  in  your  unit  to  the  training  you 
received  in  tech  school? 

2.1.1  Not  applicable;  I  didn't  attend  tech  school _ 

2. 1 .2  Not  applicable;  I  have  received  no  training  in  the  unit _ 

2.1.3  Better  in  the  unit _ 

2. 1 .4  Better  in  tech  school _ 

2. 1 .5  About  the  same _ 

2.1.6  No  opinion,  or  have  no  way  of  judging _ 

2.1  If  the  same  course  or  skill  qualification  were  being  taught  in  your  unit  and  at  a  tech 
school,  which  do  you  believe  would  better  prepare  you  to  perform  your  job? 

2.2.1  In  tech  school _  Why? 

2.2. 1 . 1  More  time  for  concentrated/detailed  training  in  tech 

school _ 

2 . 2 . 1 . 2  Tech  school  provides  a  better  teaching  environment _ 

2.2. 1 . 3  Equipment  is  more  readily  available  in  tech  school _ 

2.2. 1 .4  My  unit  has  no  one  I  feel  is  qualified  to  teach  what  I  need  to 

know _ 

2 . 2 . 1 . 5  Other  (specify) _ 

2.2.2  In  my  unit _  Why? 

2.2.2  1  Unit  level  training  is  closer  to  the  "real  world" _ 

2. 2. 2. 2  Unit  level  training  pr ,  vides  better  opportunity  for  "hands- 

on"  training _ 

2.2.2. 3  Unit  level  training  can  be  tailored  to  actual/opcrational 

equipment  or  environment _ 

2.2.2. 4  My  unit  has  trainers  who  are  better  qualified  to  teach  than 

those  at  tech  school _ 

2. 2. 2. 5  Other  (specify) _ 
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2.2.3 


They  are  both  of  equal  quality _ 

3.  TRAINING  METHODS/TECHNOLOGIES 

3.1  In  a  training  program,  a  number  of  methods  and  tools  can  be  used  such  as  overhead 
projectors,  sound-on-slidc,  videos,  computers,  etc.  I  am  going  to  read  you  a  list  of 
some  of  these.  Tell  me  which  ones  are  (or  have  been)  used  for  training  in  your 
unit. 

3.1.1  Computer  based  training  (CBT) _ 

3.1.2  Interactive  video  disks  (IVD) _ 

3.1.3  Videotapes _ 

3. 1 .4  Formal  instructor-conducted  classroom  instruction _ 

3.1.5  Self-study  books  or  other  printed  materials _ 

3. 1 .6  Simulators  or  mock-ups _ 

3.1.7  Operational  mission  equipment _ 

3.1.8  Non-operadonal  mission  equipment _ 

3.1.9  Other  (specify) _ _ 

3.1.10  Do  not  know _  (Go  to  next  section.) 
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3.2  Of  the  methods  or  tools  that  you  named,  which  ones  did  you  find  to  be  mflSI 
effective? 

3.2. 1  Computer  based  training  (CBT) _ 

3.2.2  Interactive  video  disks  (IVD) _ 

3.2.3  Videotapes _ 

3.2.4  Formal  instructor-conducted  classroom  instruction _ 

3.2.5  Self-study  books  or  other  printed  materials _ 

3.2.6  Simulators  or  mock-ups _ 

3.2.7  Operational  mission  equipment _ 

3.2.8  Non-operational  mission  equipment _ 

3.2.9  Other  (specify) _ 

3.3  Why  do  you  feel  this  (these)  is  (are)  the  most 

effective? _ 


3.4  Which  ones  did  vou  find  to  be  least  effective? 

3.4.1  Computer  based  training  (CBT) _ 

3.4.2  Interactive  video  disks  (IVD) _ 

3.4.3  Videotapes _ 

3.4.4  Formal  instructor-conducted  classroom  instruction 

3.4.5  Self-study  books  or  other  printed  materials _ 

3.4.6  Simulators  or  mock-ups _ 

3.4.7  Operational  mission  equipment _ 

3.4.8  Non-operational  mission  equipment _ 

3.4.9  Other  (specify) _ 


3.5  Why  do  you  feel  this  (these)  is  (are)  the  least 
effective? _ 


4.  TRAINING  EVALUATIONS 

4. 1  How  are  the  results  of  your  training  evaluated? 

4.1.1  Trainer  observes  my  performance  (over-the-shoulder) _ 

4.1.2  Trainer  verbally  questions  me  about  job  performance _ 

4. 1 .3  My  supervisor  is  interviewed _ 

4. 1 .4  Written  examinations _ 

4. 1 .5  Other  (specify) _ 

4.2  Are  you  satisfied  that  this  is  an  accurate  assessment  of  your  knowledge  and  skills? 

4.2.1  Yes _ 

4.2.2  No _ 

4.2.2. 1  What  would  be  a  better  assessment? 


4.2.3  Do  not  know _ _ 

4.3  Are  you  satisfied  that  your  training  evaluations  reflect  your  ability  to  perform  your 
job  tasks? 

4.3.1  Yes _ 

4.3.2  No _ 

4.3.2. 1  Whv  not?  _ 
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5.  PARTICIPATION  IN  DESERT  SHfELD/DESERT  STORM 

5. 1  Did  you  participate  in  Desert  Shield  and/or  Desert  Storm  (conflict  with  Iraq,  August 
1990  to  February  1991)? 

5.1.1  Yes _  (Go  to  DS  Interview  Form.) 

5.1.2  No _  THANK  THE  RESPONDENT 
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Appendix  C-4 

Desert  Shield/Desert  Storm 
Interview  Guide 
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AIR  FORCE  SQUADRON  LEVEL  TRAINING  PROJECT 
INDIVIDUAL  INTERVIEW 
DESERT  SHIELD/DESERT  STORM  PARTICIPANTS 


ADMINISTRATIVE  NOTE 

This  addendum  interview  is  intended  only  for  those  who  participated  in  Desen  Shield 
and/or  Desen  Storm.  The  interview  will  be  conducted  in  conjunction  with,  but  following  the 
primary  interview. 


INTRODUCTION 

I  would  now  like  to  continue  the  interview  with  a  few  additional  questions  which 
specifically  focus  on  your  participation  in  Desen  Shield  and/or  Desen  Storm. 

Again,  feel  free  to  stop  me  if  I  ask  a  question  that  is  not  clear  or  you  want  repeated.  If  any 
question  infringes  on  what  you  consider  to  be  sensitive  or  classified  information,  please  point  that 
fact  out. 

Do  you  have  any  questions? 


1.  BACKGROUND 

1 . 1  Did  you  deploy  in  suppon  of  Operation  Desen  Shield  and/or  Desen  Storm? 

1.1.1  No.  I  remained  here  (or  at  my  base  at  the  time)  in  a  suppon  or  alen 

role. _  (Go  to  next  section.) 

1.1.2  Yes _ 

1.2  To  where  were  you  deployed? 

1 .2. 1  Forward  area  in  the  Middle  East  (e.g.,  Saudi  Arabia,  Oman) _ 

1 .2.2  Rear  echelon  or  staging  area  (e.g.,  Europe) _ 

1.2.3  CONUS  base _ 

1.2.4  Other  (specify) _ 

1 .3  For  how  long  were  you  deployed? _ 

1 .4  What  was  your  primary  duty  while  deployed?. _ _ 
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.5  In  what  training  capacity  did  you  function? 

1.5.1  Training  Manager _ 

1.5.2  Trainer _ 

1.5.3  Trainee _ 

1.5.4  U  nit  Commander _ 

1.5.5  Supervisor _ 

1.5.6  None _ 

ASSESSMENT  OF  J£AMN£ 

How  well  prepared  were  you  for  the  duty  requirements  of  Desert  Shield/Des^n 
Storm? 

2.1.1  Well  prepared _ 

2. 1 .2  Not  well  prepared _ 

2. 1.2.1  Why  not?„ _ 


What  skill  or  informational  training  did  you  receive  or  provide  during  Desen 
Shield/Desen  Storm? 

2.2.1  Normal  OJT _ 

2.2.2  New  job-related  skills/information _ 

2.2.3  Living  and  working  in  foreign  country _ 

2.2.4  Additional  ancillary  training  for  threats  such  as  nuclear/chemical/biological 

warfare _ 

2.2.5  Special  training  to  learn  how  to  operate  or  maintain  equipment  in  the  desen 

environment _ 

2.2.6  Other  (specify) _ 

2.2.7  None _  (THANK  THE  RESPONDENT  AND  CONCLUDE 

THE  INTERVIEW) 


2.3  How  quickly  was  this  training  provided? 

2.3.1  Immediately _ 

2.3.2  After  some  minor  delay _ 

2.3.3  After  considerable  delay _ 

2.3.4  Too  late  to  be  of  value _ 

2.4  Who  provided  this  training _ 

2.4.1  Supervisor _ 

2.4.2  Trainer _ 

2.4.3  Field  Training  Detachment  (FTD)  Instructor _ 

2.4.4  I  provided  the  training _ 

2.4.5  Other  (specify) _ 

2.5  What  problems  were  encountered  in  conducting  the  training? 

2.5. 1  Scheduling  around  mission  requirements _ 

2.5.2  Availability  of  qualified  instructors 

2.5.3  Availability  of  up-to-date,  pertinent  training  materials  or  equipment 

2.5.4  Lack  of  adequate  training  facilities _ 

2.5.5  Other  (specify) _ 

2.5.6  None _ 

2.6  The  problems  were: 

2.6.1  Insurmountable _ 

2.6.2  Not  insurmountable;  they  just  required  the  following 

innovations: _ 
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2.7  "Normal’’  training  (e.g.,  OJT  Upgrade  Training)  was: 

2.7.1  Continued  with  no  or  minimal  interruption _ 

2.7.2  Waived  or  ignored 

2.7.3  Not  required  because  we  only  took  trained  personnel _ 

L  IBAIMNGJiffiTHQDS^CHNQLQGIES 

3. 1  What  methods  were  used  to  provide  the  training? 

3.1.1  Hands-on  and  over-the-shoulder  instruction _ 

3.1.2  Self-paced  instruction  using  written  materials,  such  as  texts  or 

manuals _ 

3.1.3  Group  stud>  of  written  materials _ 

3. 1 .4  Computer-assisted  instruction  (CAD 

3. 1 .5  Interactive  video  disks  (IVD) _ 

3.1.6  Video  cassette  recordings  (VCR) 

3.1.7  Simulators  or  mock-ups _ 

3.1.8  Lecture/presentation _ 

3. 1 .9  Other  (specify) _ 

3.2  Which  methods  worked  best? 

3.2. 1  Hands-on  and  over-the-shoulder  instruction _ 

3.2.2  Self-paced  instruction  using  written  materials,  such  as  texts  or 

manuals _ 

3.2.3  Group  study  of  written  materials _ 

3.2.4  Computer-assisted  instruction  (CAI) _ 

3.2.5  Interactive  video  disks  (IVD) _ 

3.2.6  Video  cassette  recordings  (VCR) _ 

3.2.7  Simulators  or  mock-ups _ 

3.2.8  Lecture/presentation _ 

3.2.9  Other  (specify) _ 
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3.3  Why  did  they  work  best?. 


3.4  Which  methods  were  tried,  but  did  not  work  weii? 

3.4. 1  Hands-on  and  over-the-shoulder  instruction _ 

3.4.2  Seif-paced  instruction  using  written  materials,  such  as  texts  or 

manuals _ 

3.4.3  Group  study  of  written  materials _ 

3.4.4  Computer-assisted  instruction  (CAI) _ 

3.4.5  Interactive  video  disks  (IVD) _ 

3.4.6  Video  cassette  recordings  (VCR) _ 

3.4.7  Simulators  or  mock-ups _ 

3.4.8  Lecture/prescntation _ 

3.4.9  Other  (specify) _ 

3.4.10  None _ 

3.5  Why  did  they  not  work  well? _ 
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3.6  Are  you  aware  of  any  methods  or  technologies  that  could  have  been  used  to 
enhance  training,  but  were  not? 


3.6.1  No 

3.6.2  Yes:  for  example: 

3.6.2. 1 

Hands-on  and  over-the-shoulder  instruction 

3. 6. 2. 2 

Self-paced  instruction  using  written  materials,  such  as  texts 
or  manuals 

3. 6.2. 3 

Group  studv  of  written  materials 

3. 6.2. 4 

Computer-assisted  instruction  (CA1) 

3. 6. 2. 5 

Interactive  video  disks  (IVD) 

3. 6. 2. 6 

Video  cassette  recordings  (VCR) 

3. 6.2.7 

Simulators  or  mock-ups 

3. 6. 2. 8 

Lecture/presentation 

3. 6. 2. 9 

Other  (specify) 

What  other  changes  would  you  make  in  the  manner  in  which  training  was 
conducted,  and  whv? 
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4. 1  What  do  you  consider  to  be  the  essential  training  lessons  learned  as  a  result  of  this 
operation? 


4.2  If  you  could  change  anything  done  with  regard  to  training  during  such  a  wartime 
operation,  what  would  it  be,  and 

why? _ 


THANK  THE  RESPONDENT 


Appendix  D 

Data  Analysis  Code  Sheets 


(This  appendix  provides  guidance  for  those  who  wish  to  review  or  re-analyze  the 
data  collected  for  the  Squadron  Level  Training  Phase  C  project.  Through  the  use  of 
these  code  sheets,  data  responses  to  any  specific  question  or  background  information 
can  be  located  in  the  computerized  data  set.  Separate  data  bases  exist  for  survey 
information  collected  from  Training  Managers,  Trainers,  and  Trainees.  This  is  also 
true  for  interview  information  which  also  includes  a  separate  data  base  for  Desert 
Storm/Shield  related  dataj 
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Appendix  D-l 

Training  Manager 
Survey  Code  Sheets 


251 


Sfe$g!3S8!eB8SiS«Bti6>fcj52fcSBti83sSt3KSs:CBt2S'«>«''*o'ui»wK.~ 


FLD# 


STRUCTURE  OF  TRAINER  DATA 


Field  Name 


Type  NC 


REQ.NO 

CE111 

MA1NT112 

MED113 

MSI  14 

FERS115 

SP116 

SUP117 

TRANS118 

MAC121 

SAC122 

TAC123 

AFSC13 

PAESC131 

5XXX132 

TES14 

FRD61 

ADD162 

MEASTR17 

OJT181 

UGT182 

ANQL183 

PME184 

FTD185 

RANK1101 

KILL1102 

EACH!  103 

EXPT1104 

TRTR1105 

NELSU06 

VOL1107 

osmio8 

EXP1109 

RANK11U 

KILL1112 

EACH1113 

EXFT1114 

TRTRU15 

NELS1116 

VOL1117 

osmii8 


Numeric  3 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Character  7 

Character  7 

Numeric  4 

Numeric  4 

Numeric  1 

Numeric  1 

Numeric  4 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 
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FLD#  Field  Name 

EXP1119 
NA11111 
YES1121 
NOH22 
AIC1131 
TSUP1132 
TH1133 
OJT3134 
SELF1135 
FTD1136 
LEAD1137 
EXP1138 
ONE11310 
SUFV1141 
OMDR1142 
HSSC1143 
IRST1144 
UTM1145 
ODEC1146 
QA1147 
ESUC1151 
QUAL1152 
DTA1153 
DOX1154 
NEDKII56 
OH1161 
ETNN1162 
STSN1163 
FTFN1164 
TTTN1165 
LTT1166 
N01171 
YES1172 
NOH81 
YES1182 
LC11821 
NRH11822 
DIM11823 
ENH11824 
LEX11825 
CMS1191 
SMS1192 
MSG1193 
TSG1194 
SSG1195 


Type  NC 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 
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VER0002X:  SLT  SURVEY  FORM  FOR  TRAINING  MANAGERS 


FLD#  NC  SC 

EC 

NAME 

description 

RANGE 

1 

1 

1 

1 

LISTA 

CODE  FOR  MAJ.  COMMANDS 

FIELD  OPR.  AGENCIES.  ETC 

2 

10 

2 

11 

FILLER 

FILLER 

BLANKS 

3 

1 

12 

12 

LISTB 

CODE  FOR  DESCRIPTION  OF  UNIT  OF  ASSIGNMENT 

4 

U 

13 

26 

FILLER 

FILLER 

BLANKS 

5 

9 

27 

35 

SSAN 

SOCIAL  SECURITY  ACCT  NUMBER  0-9 

6 

4 

36 

39 

VERSION 

VERSION  NUMBER 

09 

7 

12 

40 

51 

FILLER 

FILLER 

BLANKS 

8 

1 

52 

52 

Q01 

QUESTION  1 

9 

1 

53 

53 

QQ2 

QUESTION  2 

10 

1 

54 

54 

QQ3RES01 

QUESTION  3,  RESPONSE  1 

01 

11 

1 

55 

55 

QQ3RESQ2 

QUESTION  3,  RESPONSE  2 

ai 

12 

1 

56 

56 

Q03RESG3 

QUESTION  3,  RESPONSE  3 

01 

13 

1 

57 

57 

QQ3RES04 

QUESTION  3,  RESPONSE  4 

01 

14 

1 

SB 

SB 

QQ3RESG5 

QUESTION  3,  RESPONSE  5 

01 

15 

1 

59 

59 

QQ3RESQ6 

QUESTION  3,  RESPONSE  6 

01 

16 

1 

60 

60 

Q03RES07 

QUESTION  3,  RESPONSE  7 

01 

17 

1 

61 

61 

QQ3RES08 

QUESTION  3,  RESPONSE  8 

01 

18 

1 

62 

62 

Q03RES09 

QUESTION  3,  RESPONSE  9 

01 

19 

1 

63 

a 

QQ3RES10 

QUESTION  3,  RESPONSE  10 

01 

20 

1 

64 

64 

Q03RES11 

QUESTION  3,  RESPONSE  11 

01 

21 

1 

65 

65 

004 

QUESHGN4 

22 

1 

66 

66 

005 

QUESTIONS 

23 

1 

67 

67 

Q06RES01 

QUESTION  6,  RESPONSE  1 

01 

24 

1 

68 

68 

Q06RESQ2 

QUESTION  6,  RESPONSE  2 

01 

25 

1 

69 

69 

QUESTION  6,  RESPONSE  3 

01 

26 

1 

70 

70 

Q06RES04 

QUESTION  6,  RESPONSE  4 

01 

27 

1 

71 

71 

Q06RES05 

QUESTION  6,  RESPONSE  5 

01 

28 

1 

72 

72 

Q06RES06 

QUESTION  6,  RESPONSE  6 

0,1 

29 

1 

73 

73 

Q06RES07 

QUESTION  6,  RESPONSE  7 

01 

30 

1 

74 

74 

Q06RES08 

QUESTION  6,  RESPONSE  8 

01 

31 

1 

75 

75 

Q06RES09 

QUESTION  6,  RESPONSE  9 

01 

32 

1 

76 

76 

Q06RES10 

QUESTION  6,  RESPONSE  10 

Ol 

33 

1 

77 

77 

007 

QUESTION  7 

34 

1 

78 

78 

O08RES01 

QUESTION  8,  RESPONSE  1 

01 

15 

1 

79 

79 

Q08RESQ2 

QUESTION  8,  RESPONSE  2 

0,1 

16 

1 

80 

80 

Q08RES03 

QUESTION  8,  RESPONSE  3 

01 

17 

1 

81 

81 

Q08RES04 

QUESTION  8,  RESPONSE  4 

01 

e 

1 

82 

82 

Q08RES05 

QUESTION  8,  RESPONSE  5 

01 

19 

1 

83 

83 

Q08RES06 

QUESTION  8,  RESPONSE  6 

01 

10 

1 

84 

84 

QQ8RES07 

QUESTION  8,  RESPONSE  7 

01 

11 

1 

85 

85 

Q08RES08 

QUESTION  8,  RESPONSE  8 

0,1 

12 

1 

86 

86 

009 

QUESTION  9 

a 

1 

87 

87 

Q10 

QUESTION  10 

14 

1 

88 

88 

Q11RES01 

QUESTION  11,  RESPONSE  1 

0,1 

s 

1 

89 

89 

Q11RES02 

QUESTION  11,  RESPONSE  2 

01 

6 

1 

90 

90 

Q11RES03 

QUESTION  11,  RESPONSE  3 

01 

7 

1 

91 

91 

Q11RES04 

QUESTION  11,  RESPONSE  4 

0,1 

8 

1 

92 

92 

Q11RES05 

QUESTION  11,  RESPONSE  5 

0,1 

9 

1 

93 

93 

Q11RES06 

QUESTION  11,  RESPONSE  6 

0,1 

0 

1 

94 

94 

Q11RES07 

QUESTION  11,  RESPONSE  7 

01 

1 

1 

95 

95 

Q11RES08 

QUESTION  11,  RESPONSE  8 

01 

2 

1 

96 

96 

Q11RES09 

QUESTION  11,  RESPONSE  9 

0,1 

3 

1 

97 

97 

Q12 

QUESTION  12 

1-4 

A-Z 


A-Z 


A-I 

A-E 


A-E 

A.B 


A-F 


A-F 

A-D 
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FLD0  NC 

:  sc 

EC 

NAME 

DESCRIPTION 

RANGE 

54 

1 

98 

96 

Q13 

QUESTION  13 

1-5 

55 

1 

99 

99 

QM 

QUESTION  14 

1-5 

56 

1 

100 

100 

Q15 

QUESTION  15 

1-5 

57 

1 

101 

101 

Q16 

QUESTION  16 

1-5 

SB 

1 

102 

102 

017 

QUESU0N17 

1-5 

59 

1 

109 

109 

Q18 

QUESTION  18 

1-5 

60 

1 

104 

104 

Q19 

QUE5U0N19 

1-5 

61 

1 

106 

105 

Q2Q 

QUESU0N20 

62 

1 

106 

106 

Q21KES01 

QUESTION  21,  RESPONSE  1 

01 

63 

1 

107 

107 

Q21RESQ2 

QUESTION  21,  RESPONSE  2 

01 

64 

1 

108 

109 

Q21RESQ3 

QUESTION  21,  RESPONSE  3 

01 

65 

1 

109 

109 

Q21RES04 

QUESTION  21,  RESPONSE  4 

01 

66 

1 

110 

110 

Q21RES05 

QUESTION  21,  RESPONSE  5 

01 

a 

1 

111 

111 

QZUES06 

QUESTION  21,  RESPONSE  6 

01 

6B 

1 

112 

112 

fjwpBcm 

QUESTION  22,  RESPONSE  1 

01 

69 

1 

113 

US 

Q22RESQ2 

QUESTION  22,  RESPONSE  2 

01 

TO 

1 

114 

114 

Q22RES03 

QUESTION  22,  RESPONSE  3 

01 

71 

1 

115 

115 

Q22RESM 

QUESTION  22,  RESPONSE  4 

01 

72 

1 

116 

116 

QgREStB 

QUESTION  22,  RESPONSE  5 

01 

73 

1 

117 

117 

Q22EES0S 

QUESnON  22.  RESPONSE  6 

01 

74 

1 

118 

119 

Q2ZRES07 

QUESTION  22.  RESPONSE  7 

01 

75 

1 

119 

119 

Q22RESQ6 

QUESTON  22,  RESPONSE  8 

01 

76 

1 

120 

120 

Q22RES09 

QUESTION  22,  RESPONSE  9 

01 

77 

1 

121 

121 

Q22RES10 

QUESTION  22,  RESPONSE  10 

01 

78 

1 

122 

122 

Q22RESU 

QUESTION  22,  RESPONSE  11 

01 

79 

1 

123 

129 

023 

QUESTON  23 

80 

1 

124 

124 

Q24KES01 

QUESTION  24,  RESPONSE  1 

01 

81 

1 

125 

125 

Q24RESQ2 

QUESTION  24,  RESPONSE  2 

01 

82 

1 

126 

126 

Q24RES03 

QUESTION  24,  RESPONSE  3 

01 

83 

1 

127 

127 

Q24RES04 

QUESTION  24,  RESPONSE  4 

01 

84 

1 

128 

128 

Q24RES05 

QUESTION  24,  RESPONSE  5 

01 

85 

1 

129 

129 

Q24RES06 

QUESTION  24,  RESPONSE  6 

Ol 

86 

1 

130 

130 

Q24RES07 

QUESTION  24,  RESPONSE  7 

01 

87 

1 

131 

131 

Q24RES0B 

QUESTION  24,  RESPONSE  8 

01 

88 

1 

132 

132 

Q24RES09 

QUESTION  24,  RESPONSE  9 

01 

89 

1 

133 

133 

025 

QUESTION  25 

90 

1 

134 

134 

Q26 

QUESTION  26 

91 

1 

135 

135 

027 

QUESTION  27 

1-5 

92 

1 

136 

136 

Q28RE531 

QUESTION  28,  RESPONSE  1 

01 

93 

1 

137 

137 

028RESQ2 

QUESTION  28,  RESPONSE  2 

01 

94 

1 

138 

139 

Q28RES03 

QUESTION  28,  RESPONSE  3 

01 

95 

1 

139 

139 

Q28RES04 

QUESTION  28,  RESPONSE  4 

01 

96 

1 

140 

140 

Q28RES0S 

QUESTION  28,  RESPONSE  5 

0,1 

97 

1 

141 

141 

Q29RES01 

QUESTION  29,  RESPONSE  1 

0,1 

96 

1 

142 

142 

Q29RESQ2 

QUESTION  29,  RESPONSE  2 

Ol 

99 

1 

143 

143 

Q29RES03 

QUESTION  29,  RESPONSE  3 

0,1 

100 

1 

144 

144 

Q29RES04 

QUESTION  29,  RESPONSE  4 

01 

101 

1 

145 

145 

O29RES05 

QUESTION  29,  RESPONSE  5 

01 

102 

1 

146 

146 

Q29RES06 

QUESTION  29,  RESPONSE  6 

01 

103 

1 

147 

147 

Q29RES07 

QUESTION  29,  RESPONSE  7 

Ol 

104 

1 

148 

148 

Q29RES08 

QUESTION  29,  RESPONSE  8 

01 

105 

1 

149 

149 

090 

QUESITON  30 

1-5 

106 

1 

150 

150 

001 

QUESTION 31 

1-5 

107 

1 

151 

151 

0» 

QUESTION  32 

1-5 

106 

1 

152 

152 

033 

QUESTION  33 

1-5 

A-F 


A-F 


A-F 

A-C 
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FLD# 

NC 

SC 

EC 

NAME 

DESCRIPTION 

RANGE 

109 

1 

153 

153 

Q34 

QUESTION 34 

1-5 

110 

1 

154 

154 

Q35 

QUESTION  35 

111 

1 

155 

155 

Q36RES01 

QUESTION  36,  RESPONSE  1 

01 

112 

1 

156 

156 

Q36RES02 

QUESTION  36,  RESPONSE  2 

0,1 

113 

1 

157 

157 

Q36RES36 

QUESTION  36,  RESPONSE  3 

0,1 

114 

1 

158 

158 

Q36RES04 

QUESTION  36,  RESPONSE  4 

01 

115 

1 

159 

159 

Q96RES0S 

QUESTION  36,  RESPONSE  5 

01 

116 

1 

160 

160 

Q36RES06 

QUESTION  36,  RESPONSE  6 

01 

117 

1 

161 

161 

Q96RES07 

QUESTION  36,  RESPONSE  7 

01 

118 

1 

i a 

162 

Q87 

QUESTION  37 

119 

1 

163 

163 

Q38RES01 

QUESTION  30  RESPONSE  1 

01 

1  164  164  Q38RESQ2  QUESTION  38,  RESPONSE  2  0,1 

1  165  165  Q38RES03  QUESTION  38,  RESPONSE  3  0,1 

1  166  166  Q36RES04  QUESTION  38,  RESPONSE  4  01 

1  167  167  Q99RES01  QUESTION  39,  RESPONSE  1  01 

1  168  168  Q59RESQ2  QUESTION  39,  RESPONSE  2  01 

1  169  169  Q99RESQ3  QUESTION  39,  RESPONSE  3  01 

1  170  170  Q39KES04  QUESTION  39,  RESPONSE  4  01 

1  171  171  Q39RES05  QUESTION  39,  RESPONSE  5  01 

1  172  172  Q39RES06  QUESTION  39,  RESPONSE  6  01 

1  173  173  Q39RES07  QUESTION  39,  RESPONSE  7  01 

1  174  174  QIO  QUESH0N40  A-F 

1  175  175  Q41RES01  QUESTION  41,  RESPONSE  1  01 

1  176  176  Q41RES02  QUESTION  41,  RESPONSE  2  01 

1  177  177  Q41KES03  QUESTION  41,  RESPONSE  3  04 

1  178  178  Q41RES04  QUESTION  41,  RESPONSE  4  01 

1  179  179  Q41RES05  QUESnQN  41,  RESPONSE  5  01 

1  180  180  Q41RES06  QUESTION  41,  RESPONSE  6  01 

1  181  181  Q41RES07  QUESTION  41,  RESPONSE  7  01 

1  182  182  Q42RES01  QUESTION  42,  RESPONSE  1  01 

1  183  183  Q42RES02  QUESTION  42,  RESPONSE  2  0,1 

1  184  184  Q42RES03  QUESTION  42,  RESPONSE  3  01 

1  185  185  Q42RES04  QUESTION  42,  RESPONSE  4  01 

1  186  186  Q42RES05  QUESTION  42,  RESPONSE  5  01 

1  187  187  Q42RES06  QUESTION  42,  RESPONSE  6  01 

1  188  188  Q42RES07  QUESTION  42,  RESPONSE  7  01 

1  189  189  Q42RES08  QUESTION  42,  RESPONSE  8  01 

1  190  190  Q43RES01  QUESTION  43,  RESPONSE  1  01 

1  191  191  Q43RES02  QUESTION  43,  RESPONSE  2  01 

1  192  192  Q43RES03  QUESTION  43,  RESPONSE  3  01 

1  193  193  Q43RES04  QUESTION  43,  RESPONSE  4  01 

1  194  194  Q43RES05  QUESTION  43,  RESPONSE  5  01 

1  195  195  Q43RES06  QUESTION  43,  RESPONSE  6  0,1 

1  196  196  Q43RES07  QUESTION  43,  RESPONSE  7  01 

1  197  197  Q43RES06  QUESTION  43,  RESPONSE  8  01 

1  198  196  Q44RES01  QUESTION  44,  RESPONSE  1  01 

1  199  199  Q44RESQ2  QUESTION  44,  RESPONSE  2  01 

1  200  200  Q44RESQ3  QUESTION  44,  RESPONSE  3  01 

1  201  201  Q44RES04  QUESnON  44,  RESPONSE  4  0,1 

1  202  202  Q44RES05  QUESTION  44,  RESPONSE  5  01 

1  203  203  Q44RES06  QUESTION  44,  RESPONSE  6  01 

1  204  204  Q44RES07  QUESTION  44,  RESPONSE  7  01 

1  205  205  Q45RES01  QUESTION  45,  RESPONSE  1  01 

1  206  206  Q45RESQ2  QUESTION  45,  RESPONSE  2  01 

1  207  207  Q45RESQ3  QUESTION  45,  RESPONSE  3  01 
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FID#  SC  SC 

EC 

NAME 

DESCRIPTION 

RANGE 

164 

1 

208 

208 

Q45RES04 

QUESTION  45,  RESPONSE  4 

ai 

165 

1 

209 

209 

Q45RES05 

QUESTION  45,  RESPONSE  5 

0,1 

166 

1 

210 

210 

Q45RES06 

QUESTION  45,  RESPONSE  6 

0.1 

167 

1 

211 

211 

Q46RES01 

QUESTION  46,  RESPONSE  1 

0,1 

168 

1 

212 

212 

Q46RES02 

QUESTION  46,  RESPONSE  2 

0.1 

169 

1 

213 

213 

Q46RES03 

QUESTION  46,  RESPONSE  3 

ai 

17D 

1 

214 

214 

Q46RES04 

QUESTION  46,  RESPONSE  4 

on 

171 

1 

215 

215 

Q46KES05 

QUESTION  46,  RESPONSE  5 

an 

172 

« 

6 

mm  + 

216 

Q4«*ESQ6 

QUESTION  46.  RESPONSE  6 

ai 

173 

1 

217 

217 

Q47 

QUESTION  47 

1-6 

174 

1 

218 

218 

Q48 

QUESTION  48 

1-6 

175 

1 

219 

219 

Q49 

QUESTION  49 

1-6 

176 

1 

220 

220 

(350 

QUESTION 50 

1-5 

177 

1 

221 

221 

051 

QUESTION  51 

1-6 

178 

1 

222 

222 

052 

QUESTION  52 

1-6 

179 

1 

223 

223 

053 

QUESTION  53 

1-5 

180 

1 

224 

224 

054 

QUESTION  54 

1-6 

181 

1 

225 

225 

055 

QUESTION  55 

1-5 

182 

1 

226 

226 

056 

QUESI1GN56 

1-5 

183 

1 

227 

227 

057 

QUESTION  57 

184 

1 

228 

728 

Q5KRES01 

QUESTION  58,  RESPONSE  1 

ai 

185 

1 

229 

229 

Q58RES02 

QUESTION  58,  RESPONSE  2 

01 

186 

1 

230 

230 

Q58RES03 

QUESTION  58,  RESPONSE  3 

Ol 

187 

1 

231 

231 

Q5SRES04 

QUESTION  58,  RESPONSE  4 

Ol 

188 

1 

232 

232 

Q58RES0S 

QUESTION  58,  RESPONSES 

Ol 

189 

1 

233 

233 

Q58RES06 

QUESTION  58,  RESPONSE  6 

Ol 

190 

1 

234 

234 

Q58RE907 

QUESTION  58,  RESPONSE  7 

Ol 

191 

1 

235 

235 

rywgFqift 

QUESTION  58,  RESPONSE  8 

Ol 

192 

1 

236 

236 

Q58RESQ9 

QUESTION  58,  RESPONSE  9 

01 

193 

1 

237 

237 

059 

QUESTION 59 

194 

1 

238 

238 

Q6QRES01 

QUESTION  60,  RESPONSE  1 

Ol 

195 

1 

239 

239 

Q60RES02 

QUESTION  60,  RESPONSE  2 

Ol 

196 

1 

240 

240 

Q6GRES60 

QUESTION  60,  RESPONSE  3 

01 

197 

1 

241 

241 

Q60RES04 

QUESTION  60,  RESPONSE  4 

01 

196 

1 

242 

242 

Q60RES05 

QUESTION  60,  RESPONSE  5 

Ol 

199 

1 

243 

243 

Q60RES06 

QUESTION  60,  RESPONSE  6 

01 

200 

1 

244 

244 

Q60RES07 

QUESTION  60,  RESPONSE  7 

Ol 

201 

1 

245 

245 

Q60RES08 

QUESTION  60,  RESPONSE  8 

Ol 

202 

1 

246 

246 

Q60RES09 

QUESTION  60,  RESPONSE  9 

Ol 

203 

1 

247 

247 

Q61 

QUESTION  61 

204 

1 

248 

248 

Q62 

QUESTION  62 

206 

1 

249 

249 

Q63RES01 

QUESTION  63,  RESPONSE  I 

01 

206 

1 

250 

250 

Q63RES02 

QUESTION  63,  RESPONSE  2 

Ol 

207 

1 

251 

251 

Q63RES03 

QUESTION  63,  RESPONSE  3 

01 

206 

1 

252 

252 

Q63RES04 

QUESTION  63,  RESPONSE  4 

Ol 

209 

1 

253 

253 

Q63RES06 

QUESTION  63,  RESPONSE  5 

Ol 

210 

1 

254 

254 

Q63RES06 

QUESTION  63,  RESPONSE  6 

Ol 

211 

1 

255 

255 

Q63RES07 

QUESTION  63,  RESPONSE  7 

01 

212 

1 

256 

256 

Q64RES01 

QUESTION  64,  RESPONSE  1 

Ol 

213 

1 

257 

257 

Q64RES02 

QUESTION  64,  RESPONSE  2 

01 

214 

1 

258 

258 

Q64RES03 

QUESTION  64,  RESPONSE  3 

01 

215 

1 

259 

259 

Q64RES04 

QUESTION  64,  RESPONSE  4 

01 

216 

1 

260 

260 

Q64RE505 

QUESTION  64,  RESPONSE  5 

01 

217 

1 

261 

261 

Q64RES06 

QUESTION  64,  RESPONSE  6 

Ol 

218 

1 

262 

262 

Q64RES07 

QUESTION  64,  RESPONSE  7 

Ol 

A-C 


A-I 


A-E 

A-E 


258 


3339SS!i588*gKggS8g802BBSggg§S;S£§ggggBI38BBBBSg8Si!36!HBBBB8S 


ELD#  NC  SC  EC  NAME 


DESCRIPTION 


RANGE 


1 

263 

263 

Q65 

QUESTION  65 

1 

264 

264 

066 

QUESTION  66 

1 

265 

265 

Q67 

QUESTION  67 

1 

266 

266 

Q68 

QUESTION  68 

1 

267 

267 

Q69 

QUESTION  69 

1 

268 

268 

Q7QRES0I 

QUESTION  70/  RESPONSE  1 

0,1 

1 

269 

269 

Q7DRESQ2 

QUESTION  70/  RESPONSE  2 

04 

1 

27D 

270 

Q70RESQ3 

QUESTION  70,  RESPONSE  3 

ai 

1 

271 

271 

Q7BRES04 

QUESTION  70,  RESPONSE  4 

ai 

1 

272 

272 

Q7DKES05 

QUESTION  70,  RESPONSE  5 

04 

1 

273 

273 

on 

QUESTION  71 

1 

274 

274 

Q72RES01 

QUESTION  72,  RESPONSE  1 

04 

1 

275 

275 

Q72RES02 

QUESTION  72,  RESPONSE  2 

04 

1 

276 

276 

Q72RESQ3 

QUESTION  72.  RESPONSE  3 

04 

1 

277 

277 

Q72RES04 

QUESTION  72.  RESPONSE  4 

04 

1 

278 

278 

Q72RES05 

QUESTION  72,  RESPONSE  5 

04 

1 

279 

279 

Q72RES06 

QUESTION  72,  RESPONSE  6 

04 

1 

280 

280 

Q72RES07 

QUESTION  72,  RESPONSE  7 

04 

1 

281 

281 

073 

QUESTION  73 

1 

282 

282 

Q74RES01 

QUESTION  74.  RESPONSE  1 

04 

1 

283 

283 

Q74RES02 

QUESTION  74,  RESPONSE  2 

04 

1 

284 

284 

Q74RESG3 

QUESTION  74,  RESPONSE  3 

04 

1 

285 

285 

Q74RES04 

QUESTION  74,  RESPONSE  4 

04 

1 

286 

286 

Q74RES05 

QUESTION  74,  RESPONSE  5 

0,1 

1 

287 

287 

Q74RES06 

QUESTION  74,  RESPONSE  6 

04 

1 

288 

288 

Q74RES07 

QUESTION  74,  RESPONSE  7 

04 

1 

289 

289 

Q75RES01 

QUESTION  75,  RESPONSE  1 

04 

1 

290 

290 

Q75RES02 

QUESTION  75,  RESPONSE  2 

04 

1 

291 

291 

Q75RESQ3 

QUESTION  75,  RESPONSE  3 

0,1 

1 

292 

292 

Q75RES04 

QUESTION  75,  RESPONSE  4 

04 

1 

293 

293 

Q75RES05 

QUESTION  75,  RESPONSE  5 

ai 

1 

294 

294 

Q75RES06 

QUESTION  75,  RESPONSE  6 

04 

1 

295 

295 

Q75RES07 

QUESTION  75,  RESPONSE  7 

04 

1 

296 

296 

Q76RES01 

QUESTION  76,  RESPONSE  1 

04 

I 

297 

297 

Q76RESQ2 

QUESTION  76,  RESPONSE  2 

04 

1 

296 

296 

Q76RESQ3 

QUESTION  76,  RESPONSE  3 

04 

1 

299 

299 

Q76RES04 

QUESTION  76,  RESPONSE  4 

0,1 

1 

300 

300 

Q76RES05 

QUESTION  76,  RESPONSE  5 

04 

1 

301 

301 

Q76RES06 

QUESTION  76,  RESPONSE  6 

0,1 

1 

302 

302 

Q/6RES07 

QUESTION  76,  RESPONSE  7 

ai 

1 

303 

303 

Q77RES01 

QUESTION  77,  RESPONSE  1 

ai 

I 

304 

304 

Q77RES02 

QUESTION  77,  RESPONSE  2 

ai 

1 

305 

305 

Q77RES03 

QUESTION  77,  RESPONSE  3 

ai 

1 

306 

306 

Q77RES04 

QUESTION  77,  RESPONSE  4 

0,1 

1 

307 

307 

Q77RES05 

QUESTION  77,  RESPONSE  5 

0,1 

1 

306 

306 

Q77RES06 

QUESTION  77,  RESPONSE  6 

04 

1 

309 

309 

Q77RES07 

QUESTION  77,  RESPONSE  7 

ai 

1 

310 

310 

Q7BRES01 

QUESTION  78,  RESPONSE  1 

ai 

1 

311 

311 

Q7BRESQ2 

QUESTION  78,  RESPONSE  2 

ai 

1 

312 

312 

Q78RESQ3 

QUESTION  78,  RESPONSE  3 

ai 

1 

313 

313 

Q78RES04 

QUESTION  78,  RESPONSE  4 

ai 

1 

314 

314 

Q78RES05 

QUESTION  78,  RESPONSE  5 

ai 

1 

315 

315 

Q78RES06 

QUESTION  78,  RESPONSE  6 

ai 

I 

316 

316 

Q78RES07 

QUESTION  78,  RESPONSE  7 

0,1 

1 

317 

317 

Q79RES01 

QUESTION  79,  RESPONSE  1 

ai 

A-C 

A-C 

A-F 

A-F 

A-F 


A-D 
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FLD#  SC 

SC 

BC 

NAME 

DESCRIPTION 

RANGE 

274  1 

316 

318 

Q79RESQ2 

QUESTION  79,  RESPONSE  2 

03 

275  1 

319 

319 

Q79RESQ3 

QUESTION  79,  RESPONSE  3 

ftl 

276  1 

320 

320 

Q79KES04 

QUESTION  79,  RESPONSE  4 

0,1 

277  1 

321 

321 

Q79RES05 

QUESTION  79,  RESPONSE  5 

ftl 

278  1 

322 

322 

Q79RES06 

QUESTION  79,  RESPONSE  6 

0,1 

279  1 

323 

323 

Q79RJES07 

QUESTION  79,  RESPONSE  7 

OH 

280  1 

334 

324 

Q80RES01 

QUESTION  80,  RESPONSE  1 

OH 

281  1 

325 

325 

QB0RES02 

QUESTION  80,  RESPONSE  2 

ai 

282  1 

326 

326 

Q80RES03 

QUESTION  8Q,  RESPONSE  3 

ai 

2D  1 

327 

327 

Q6QRES04 

QUESTION  80,  RESPONSE  4 

ai 

281  1 

328 

328 

Q6QRESQ5 

QUESTION  80,  RESPONSE  5 

ai 

2D  1 

329 

3D 

Q80RES06 

QUESTION  80,  RESPONSE  6 

oh 

286  1 

330 

330 

Q60RES07 

QUESTION  80,  RESPONSE  7 

fti 

287  2 

331 

332 

MAJOOM  ASGT-CURRENT-MAJOR  COMMAND  ID 

A-Z 

2D  6 

333 

338 

PAfSC  AFSC- PRIMARY -NUMHER/5UFFIX  0-9, 

AC 

2D  6 

3D 

344 

DAPSC  AH9C-  DUTY -NUMBBR/SUFFDC 

0*9, 

AC 

290  17 

345 

361 

DTYLOC  ASGT- DUTY  LOCATION  NAME 

A-Z 

291  2 

362 

363 

GRADE  CURRENT  CRAEG 

0-9 

260 


Appendix  D-2 

Trainer  Survey 
Code  Sheets 
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VER0003X:  SLT  SURVEY  FORM  FOR  UNIT-LEVEL  TRAINERS 


FLO#  NC 

SC 

EC 

NAME  DESCRIPTION 

RANGE 

1 

1 

1 

1 

LISTA  CODE  FOR  MAJ.  COMMANDS 

HELD  OPR.  AGENCIES.  ETC. 

A-Z 

2 

10 

2 

11 

FILLER  FILLER 

BLANKS 

3 

1 

12 

12 

USTB  CODE  FOR  DESCRIPTION  OF  UNfT  OF  ASSIGNMENT 

A-Z 

4 

14 

13 

26 

FILLER  FILLER 

blanks 

6 

9 

27 

35 

SSAN  SOCIAL  SECURITY  ACCT  NUMBER  0-9 

6 

4 

36 

39 

VERSION  VERSION  NUMBER 

06 

7 

12 

40 

51 

FILLER  FILLER 

BLANKS 

8 

1 

52 

52 

Q01  QUESTION  1 

M 

9 

1 

53 

53 

002  QUESTION  2 

A-E 

10 

1 

54 

64 

Q03RES01  QUESTION  3,  RESPONSE  1 

0.1 

11 

1 

55 

65 

Q03RES02  QUESTION  3,  RESPONSE  2 

0,1 

12 

1 

56 

56 

Q03RES03  QUESTION  3.  RESPONSE  3 

0.1 

13 

1 

67 

57 

Q03RES04  QUESTION  3.  RESPONSE  4 

0.1 

14 

1 

58 

58 

Q03RES05  QUESTION  3.  RESPONSE  5 

0.1 

15 

1 

69 

59 

Q03RES06  QUESTION  3.  RESPONSE  6 

0.1 

16 

1 

60 

60 

004  QUESTION  4 

A-L 

17 

1 

61 

61 

Q05RES01  QUESTION  5.  RESPONSE  1 

0.1 

18 

1 

62 

62 

Q05RES02  QUESTION  5.  RESPONSE  2 

0.1 

19 

1 

63 

63 

Q05RES03  QUESTION  5.  RESPONSE  3 

0.1 

20 

1 

64 

64 

Q05RES04  QUESTION  5.  RESPONSE  4 

0.1 

21 

1 

65 

65 

Q05RES05  QUESTION  5.  RESPONSE  5 

0.1 

22 

1 

66 

66 

Q05RES06  QUESTION  5.  RESPONSE  6 

0.1 

23 

1 

67 

67 

Q06  QUESTION  6 

A-F 

24 

1 

68 

68 

Q07  QUESTION  7 

AO 

25 

1 

69 

69 

Q08RES01  QUESTION  8,  RESPONSE  1 

0.1 

26 

1 

70 

70 

Q08RES02  QUESTION  8.  RESPONSE  2 

0.1 

27 

1 

71 

71 

Q08RES03  QUESTION  8.  RESPONSE  3 

0.1 

28 

1 

72 

72 

Q08RES04  QUESTION  8.  RESPONSE  4 

0.1 

29 

1 

73 

73 

Q08RES05  QUESTION  8.  RESPONSE  5 

0,1 

30 

1 

74 

74 

Q08RES06  QUESTION  8.  RESPONSE  6 

0,1 

31 

1 

75 

75 

Q09RES01  QUESTION  9,  RESPONSE  1 

0.1 

32 

1 

76 

76 

Q09RES02  QUESTION  9,  RESPONSE  2 

0.1 

33 

1 

77 

77 

Q09RES03  QUESTION  9,  RESPONSE  3 

0.1 

34 

1 

78 

78 

Q09RES04  QUESTION  9,  RESPONSE  4 

0.1 

35 

1 

79 

79 

Q09RES05  QUESTION  9,  RESPONSE  5 

0.1 

36 

1 

80 

80 

Q09RES06  QUESTION  9.  RESPONSE  6 

0.1 

37 

1 

61 

81 

Q09RES07  QUESTION  9.  RESPONSE  7 

0.1 

38 

1 

82 

82 

Q09RES08  QUESTION  9.  RESPONSE  8 

0,1 

39 

1 

83 

83 

Q09RES09  QUESTION  9.  RESPONSE  9 

0.1 

40 

1 

84 

84 

Q09RES10  QUESTION  9,  RESPONSE  10 

0.1 

41 

1 

85 

85 

Q10  QUESTION  10 

A-E 

42 

1 

86 

86 

Q11  QUESTION  11 

1-4 

43 

1 

87 

87 

Q12  QUESTION  12 

1-4 

44 

1 

88 

88 

Q13  QUESTION  13 

1-4 

45 

1 

89 

89 

Q14  QUESTION  14 

1-4 

46 

1 

90 

90 

Q16  QUESTION  15 

1-4 

47 

1 

91 

91 

Q16  QUESTION  16 

1-4 

48 

1 

92 

92 

Q17  QUESTION  17 

1-4 

49 

1 

93 

93 

Q18  QUESTION  18 

1-4 

50 

1 

94 

94 

Q19  QUESTION  19 

1-4 

51 

1 

95 

95 

Q20  QUESTION  20 

1-4 

52 

1 

96 

96 

Q21  QUESTION  21 

A-F 

S3 

1 

97 

97 

Q22RES01  QUESTION  22,  RESPONSE  1 

0,1 

263 


FLO*  NC 

8C 

EC 

NAME 

DESCRIPTION 

RANGE 

64 

1 

38 

98 

Q22RES02  QUESTION  22.  RESPONSE  2 

0,1 

55 

1 

99 

99 

Q22RES03  QUESTION  22.  RESPONSE  3 

0,1 

56 

1 

100 

100 

i  Q22RES04  QUESTION  22.  RESPONSE  4 

0.1 

57 

1 

101 

101 

Q22RES05  QUESTION  22.  RESPONSE  5 

0.1 

56 

1 

102 

102 

Q22RES06  QUESTION  22.  RESPONSE  6 

0.1 

50 

1 

103 

103 

i  Q23RES01 

1  QUESTION  23,  RESPONSE  1 

0.1 

60 

1 

104 

104 

Q23RES02 

QUESTION  23.  RESPONSE  2 

0,1 

61 

1 

106 

106 

Q23RES03 

QUESTION  23,  RESPONSE  3 

0.1 

62 

1 

106 

106 

Q23RES04 

QUESTION  23,  RESPONSE  4 

0.1 

63 

1 

107 

107 

Q23RES06 

QUESTION  23.  RESPONSE  5 

0.1 

64 

1 

106 

106 

Q23RES06 

QUESTION  23.  RESPONSE  6 

0.1 

86 

1 

109 

108 

Q23RES07 

QUESTION  23,  RESPONSE  7 

0.1 

66 

1 

110 

110 

Q23RES08 

QUESTION  23.  RESPONSE  8 

0,1 

67 

1 

111 

111 

Q23RES09 

QUESTION  23.  RESPONSE  9 

0.1 

66 

1 

112 

112 

Q23RES10 

QUESTION  23.  RESPONSE  10 

0.1 

60 

1 

113 

113 

Q23RES11 

QUESTION  23,  RESPONSE  11 

0.1 

70 

1 

114 

114 

Q23RES12 

QUESTION  23.  RESPONSE  12 

0.1 

71 

1 

115 

116 

024 

QUESTION  24 

72 

1 

116 

116 

Q25RES01 

QUESTION  25.  RESPONSE  1 

0.1 

73 

1 

117 

117 

Q25RES02 

QUESTION  25.  RESPONSE  2 

0.1 

74 

1 

118 

118 

Q25RES03 

QUESTION  25.  RESPONSE  3 

0,1 

75 

1 

119 

119 

Q25RES04 

QUESTION  25.  RESPONSE  4 

0.1 

76 

1 

120 

120 

Q25RES05 

QUESTION  25.  RESPONSE  5 

0,1 

77 

1 

121 

121 

Q25RES06 

QUESTION  25.  RESPONSE  6 

0.1 

78 

1 

122 

122 

Q25RES07 

QUESTION  25,  RESPONSE  7 

0,1 

78 

1 

123 

123 

Q25RES08  QUESTION  25.  RESPONSE  8 

0.1 

80 

1 

124 

124 

Q25RES09 

QUESTION  25.  RESPONSE  8 

0.1 

81 

1 

125 

125 

Q26 

QUESTION  25 

82 

1 

126 

126 

Q27 

QUESTION  27 

83 

1 

127 

127 

COB 

QUESTION  2B 

84 

1 

128 

128 

029 

QUESTION  29 

85 

1 

129 

129 

Q30 

QUESTION  X 

86 

1 

130 

130 

031 

QUESTION  31 

87 

1 

131 

131 

Q32 

QUESTION  32 

88 

1 

132 

132 

033 

QUESTION  33 

89 

1 

133 

133 

034 

QUESTION  34 

SO 

1 

134 

134 

035 

QUESTION 35 

91 

1 

135 

135 

036 

QUESTION  36 

1*7 

92 

1 

136 

136 

037 

QUESTION  37 

1*7 

93 

1 

137 

137 

036 

QUESTON36 

1-7 

94 

1 

138 

138 

039 

QUESTION  39 

1-7 

96 

1 

139 

139 

040 

QUESTION  40 

1-7 

96 

1 

140 

140 

041 

QUESTION  41 

1*7 

97 

1 

141 

141 

042 

QUESTION  42 

1-7 

96 

1 

142 

142 

043 

QUESTION  43 

1-7 

99 

1 

143 

143 

044 

QUESTION  44 

1-6 

100 

1 

144 

144 

045 

QUESTION  46 

1-6 

101 

1 

145 

145 

046 

QUESTION  46 

1-6 

102 

1 

146 

146 

047 

QUESTION  47 

1-6 

103 

1 

147 

147 

048 

QUESmON48 

1-6 

104 

1 

148 

148 

049 

QUESTION  48 

1-6 

106 

1 

149 

149 

050 

QUESTION  50 

1-6 

106 

1 

150 

150 

051 

QUESTION  51 

1-6 

107 

1 

151 

151 

052 

QUESTION  52 

1-6 

108 

1 

152 

152 

053 

QUESTION  S3 

1-6 

264 


FLDf  NC 

:  sc 

EC 

NAME 

DESCRIPTION 

RANGE 

100 

1 

153 

153 

Q54 

QUESTION  54 

1-6 

110 

1 

154 

154 

055 

QUESTION  55 

1-6 

111 

1 

155 

155 

066 

QUESTION  56 

1-6 

112 

1 

156 

156 

057 

QUESTION  57 

1-6 

113 

1 

157 

157 

058 

QUESTION  SB 

1-6 

114 

1 

158 

156 

059 

QUESTION  59 

1-6 

115 

1 

150 

150 

060 

QUESTION  60 

1-6 

118 

1 

180 

160 

OBI 

QUESTION  61 

1-5 

117 

1 

161 

161 

062 

QUESTION  62 

1-6 

118 

1 

162 

162 

063 

QUESTION  63 

1-6 

119 

1 

163 

163 

Q64RES01 

QUESTION  64,  RESPONSE  1 

0.1 

120 

1 

164 

164 

Q64RES02 

QUESTION  64.  RESPONSE  2 

0.1 

121 

1 

166 

166 

Q64RES03 

QUESTION  64.  RESPONSE  3 

0.1 

122 

1 

166 

166 

064RES04 

QUESTION  64.  RESPONSE  4 

0,1 

123 

1 

167 

167 

064RES05 

QUESTION  64.  RESPONSE  5 

0,1 

124 

1 

168 

168 

Q64RES06 

QUESTION  64,  RESPONSE  6 

0.1 

125 

1 

160 

160 

Q64RES07 

QUESTION  64.  RESPONSE  7 

0,1 

126 

1 

170 

170 

Q64RESQ8 

QUESTION  64,  RESPONSE  8 

0.1 

127 

1 

171 

171 

Q64RES09 

QUESTION  64.  RESPONSE  9 

0.1 

128 

1 

172 

172 

O64RES10 

QUESTION  64,  RESPONSE  10 

0,1 

129 

1 

173 

173 

065 

QUESTION  65 

130 

1 

174 

174 

Q66RES01 

QUESTION  66.  RESPONSE  1 

0,1 

131 

1 

175 

175 

Q66RES02 

QUESTION  66,  RESPONSE  2 

0.1 

132 

1 

176 

176 

Q66RES03 

QUESTION  68,  RESPONSE  3 

0.1 

133 

1 

177 

177 

Q66RES04 

QUESTION  66,  RESPONSE  4 

0.1 

134 

1 

178 

178 

Q66RES05 

QUESTION  66,  RESPONSE  5 

0.1 

135 

1 

170 

170 

Q66RES06 

QUESTION  66,  RESPONSE  6 

0,1 

136 

1 

180 

180 

Q66RES07 

QUESTION  66.  RESPONSE  7 

0.1 

137 

1 

181 

181 

Q66RES08 

QUESTION  66.  RESPONSE  8 

0,1 

138 

1 

182 

182 

Q66RES09 

QUESTION  66,  RESPONSE  0 

0.1 

139 

1 

183 

183 

Q66RES10 

QUESTION  66,  RESPONSE  10 

0.1 

140 

1 

184 

184 

067 

QUESTION  67 

141 

1 

185 

185 

068 

QUESTION  6B 

142 

1 

186 

186 

Q69RES01 

QUESTION  69,  RESPONSE  1 

0.1 

143 

1 

187 

187 

Q69RESQ2 

QUESTION  69.  RESPONSE  2 

0.1 

144 

1 

188 

188 

Q69RES03 

QUESTION  69,  RESPONSE  3 

0.1 

145 

1 

189 

180 

Q69RES04 

QUESTION  69,  RESPONSE  4 

0.1 

146 

1 

100 

100 

Q69RES05 

QUESTION  69.  RESPONSE  5 

0.1 

147 

1 

191 

191 

Q69RES06 

QUESTION  69,  RESPONSE  6 

0.1 

148 

1 

192 

102 

Q69RES07 

QUESTION  69,  RESPONSE  7 

0,1 

149 

1 

193 

193 

Q70RES01 

QUESTION  70,  RESPONSE  1 

0,1 

150 

1 

194 

194 

Q70RES02  QUESTION  70,  RESPONSE  2 

0,1 

151 

1 

195 

195 

Q70RES03  QUESTION  70,  RESPONSE  3 

0,1 

152 

1 

196 

196 

Q70RES04  QUESTION  70,  RESPONSE  4 

0.1 

153 

1 

197 

197 

Q70RES05  QUESTION  70,  RESPONSE  5 

0.1 

154 

1 

198 

198 

Q70RES06  QUESTION  70.  RESPONSE  6 

0,1 

155 

1 

199 

199 

Q70RES07  QUESTION  70,  RESPONSE  7 

0,1 

156 

1 

200 

200 

071 

QUESTION  71 

157 

1 

201 

201 

072 

QUESTION  72 

158 

1 

202 

202 

073 

QUESTION  73 

159 

1 

203 

203 

Q74RES01 

QUESTION  74,  RESPONSE  1 

0.1 

160 

1 

204 

204 

Q74RES02 

QUESTION  74,  RESPONSE  2 

0,1 

161 

1 

205 

205 

Q74RES03 

QUESTION  74,  RESPONSE  3 

0,1 

162 

1 

206 

206 

Q74RES04 

QUESTION  74,  RESPONSE  4 

0,1 

163 

1 

207 

207 

Q74RES06 

QUESTION  74,  RESPONSE  5 

0.1 

A-J 


A-D 

AO 


AG 

AG 

A-F 


265 


FLOf  NC  8 C  EC  NAME  DESCRIPTION 


RANGE 


164  1  208  208  Q74RES06  QUESTION  74,  RESPONSE  6  0.1 

1— ,  i  208  208  075  QUESTION  75 

1Ui  i  210  210  Q76RES01  QUESTION  76.  RESPONSE  1  0.1 

167  1  211  211  Q76RESQ2  QUESTION  76.  RESPONSE  2  0,1 

168  1  212  212  Q76RES03  QUESTION  76.  RESPONSE  3  0.1 

168  1  213  213  Q76RES04  QUESTION  76,  RESPONSE  4  0.1 

170  1  214  214  Q76RES05  QUESTION  76.  RESPONSE  6  0.1 

171  1  216  216  Q77  QLEST10N77  A-D 

172  1  216  218  Q78RES01  QUESTION  78.  RESPONSE  1  0,1 

173  1  217  217  Q78RES02  QUESTION  78.  RESPONSE  2  0.1 

174  1  218  218  O78RES03  QUESTION  78,  RESPONSE  3  0.1 

175  1  218  218  Q78RES04  QUESTION  78.  RESPONSE  4  0.1 

176  1  220  220  Q78RES05  QUESTION  78,  RESPONSE  6  0.1 

177  1  221  221  Q78RES06  QUESTION  78.  RESPONSE  6  0.1 

178  1  222  222  Q78RES07  QUESTION  78,  RESPONSE  7  0,1 

179  1  223  223  Q7BRES06  QUESTION  78.  RESPONSE  8  0.1 

180  1  224  224  078  QUESTION  79  A-F 

181  1  225  225  Q30RES01  QUESTION  80,  RESPONSE  1  0.1 

182  1  228  226  Q80RESQ2  QUESTION  80.  RESPONSE  2  0.1 

183  1  227  227  Q80RES03  QUESTION  80.  RESPONSE  3  0.1 

184  1  228  228  Q60RES04  QUESTION  80,  RESPONSE  4  0.1 

185  1  229  229  Q80RES06  QUESTION  80.  RESPONSE  6  0.1 

186  1  230  230  OBORESOQ  QUESTION  80.  RESPONSE  6  0.1 

187  1  231  231  Q80RES07  QUESTION  80.  RESPONSE  7  0.1 


188  1  232  232  O81RES01  QUESTION  81.  RESPONSE  1  0.1 

189  1  233  233  Q81RES02  QUESTION  81.  RESPONSE  2  0.1 

190  1  234  234  Q81RES03  QUESTION  81 .  RESPONSE  3  0.1 

191  1  235  235  Q81RES04  QUESTION  81.  RESPONSE  4  ft', 

192  1  236  236  Q81RESG5  QUESTION  81.  RESPONSE  5  0.1 

183  1  237  237  QB1RES06  QUESTION  81.  RESPONSE  6  0.1 

194  1  238  238  QB1RES07  QUESTION  81 .  RESPONSE  7  0.1 


195  1  239  239  QS2RES01  QUESTION  82,  RESPONSE  1  0.1 

196  1  240  240  Q82RES02  QUESTION  82,  RESPONSE  2  0.1 

197  !  241  241  O82RES03  QUESTION  82.  RESPONSE  3  0.1 

198  1  242  242  CH12RES04  QUESTION  82.  RESPONSE  4  0.1 

199  1  243  243  Q82RES05  QUESTION  82.  RESPONSE  5  0.1 

200  1  244  244  Q82RES06  QUESTION  82.  RESPONSE  6  0.1 


201  1  245  245  082  ~07  QUESTION  82.  RESPONSE  7  0.1 

202  1  246  246  Q83Wt  31  QUESTION  83.  RESPONSE  1  0.1 

203  1  24?  247  O83RES0;  QUESTION  83,  RESPONSE  2  0.1 

204  1  24o  248  O83RES03  QUESTION  83,  RESPONSE  3  0,1 

205  1  249  249  O83RES04  QUESTION  83.  RESPONSE  4  0.1 

206  1  250  250  Q*3RES05  QUESTION  83.  RESPONSE  5  0,1 

207  1  251  25.  Q&3RES06  QUESTION  83.  RESPONSE  6  0.1 

208  1  252  252  Q83HESQ7  QUESTION  83.  RESPONSE  7  0.1 

200  1  253  253  O84RES01  QUESTION  84.  RESPONSE  1  0.1 

210  1  254  254  Q84P.ES0?  QUESTION  84,  RES°ONSE  2  0,1 

211  1  255  255  084RES01  QUESTION  84,  RESPONSE  3  0.1 

212  1  256  256  G84RE804  QUESTION  84,  RESPONSE  4  0.1 

213  1  257  257  QMRES05  QUESTION  84  RESPONSE  5  0.1 

214  1  258  258  O84RES0«  QUESTION  84.  RESPONSE  6  0.1 

215  1  259  259  084  HE  SOT  QUESTION  84.  RESPONSE  7  0.1 

216  1  260  760  QaSRESOi  QUESTION 85,  RESPONSE  1  0,1 

217  1  261  261  QeSRESG?  QUESTION  85  RESPONSE  2  0,1 

218  1  262  262  Q85RES03  QUESTION  85.  RESPONSE  3  0.1 


FLD#  NC 

8C  EC 

NAME 

DESCRIPTION 

range 

210 

1 

263  263 

Q85RES04 

QUESTION  85.  RESPONSE  4 

0.1 

220 

1 

264  264 

Q85RES05 

QUESTION  85.  RESPONSE  5 

0,1 

221 

1 

266  265 

Q85RES06 

QUESTION  85,  RESPONSE  6 

0.1 

222 

1 

266  266 

Q85RES07 

QUESTION  85,  RESPONSE  7 

0,1 

223 

1 

267  267 

Q86RES01 

QUESTION  86,  RESPONSE  1 

0.1 

224 

1 

268  268 

Q86RES02 

QUESTION  86,  RESPONSE  2 

0,1 

225 

1 

260  269 

Q86RES03 

QUESTION  86,  RESPONSE  3 

0.1 

225 

1 

270  270 

Q86RES04 

QUESTION  86.  RESPONSE  4 

0.1 

227 

1 

271  271 

Q66RES06 

QUESTION  86,  RESPONSE  5 

0.1 

228 

1 

272  272 

Q86RES06 

QUESTION  86.  RESPONSE  6 

0.1 

220 

1 

273  273 

Q66RES07 

QUESTION  86,  RESPONSE  7 

0.1 

230 

1 

274  274 

Q67RES01 

QUESTION  87,  RESPONSE  1 

0.1 

231 

1 

275  275 

Q67RES02 

QUESTION  87,  RESPONSE  2 

0.1 

232 

1 

276  278 

Q87RES03 

QUESTION  87.  RESPONSE  3 

0.1 

233 

1 

277  277 

Q87RES04 

QUESTION  87.  RESPONSE  4 

0.1 

234 

1 

278  278 

Q67RES05 

QUESTION  87.  RESPONSE  S 

0.1 

235 

1 

279  279 

Q87RES06 

QUESTION  87.  RESPONSE  6 

0.1 

236 

1 

280  280 

Q87RES07 

QUESTION  87.  RESPONSE  7 

0.1 

237 

1 

281  281 

QBB 

QUESTION  88 

238 

2 

282  283 

ASGT  -  CURRENT  •  MAJOR  COMMAND  ID 

239 

6 

284  289 

PAFSC 

AFSC  -  PRIMARY  -  NUMBER/SUFFIX  0-9, 

240 

6 

290  295 

DAFSC 

AFSC  -  DUTY  -  NUMBER/SUFFIX 

0*9. 

241 

17 

296  312 

DTYLOC 

ASGT  -  DUTY  LOCATION  NAME 

242 

2 

313  314 

GRADE 

CURRENT  GRADE 

09 

A-D 

A-Z 

A-C 

AC 

A-Z 


267 


Appendix  D-3 

Trainee  Survey 
Code  Sheets 


269 


VER0004X:  SLT  SURVEY  FORM  FOR  RECIPIENTS  OF  TRAINING 


FLO#  NC  8C  EC  NAME  DESCRIPTION 


RANGE 


1 

2 

3 

4 
6 
6 

7 

8 

9 

10 
11 
12 

13 

14 
16 
16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 


1 

1 

1 

USTA 

CODE  FOR  MAJOR  COMMANDS.FIELD  OPR.  AGENCIES.  ETC. 

10 

2 

11 

FILLER 

FILLER 

BLAN1S.O 

1 

12 

12 

LISTB 

CODE  FOR  DESCRIPTION  OF  UNTT  OF  ASSIGNMENT 

14 

13 

26 

FILLER 

FILLER 

BLANKS 

9 

27 

36 

SSAN 

SOCIAL  SECURITY  ACCOUNT  NUMBER 

09 

4 

36 

39 

VERSION 

VERSION  NUMB® 

0-9 

12 

40 

51 

FILLER 

RIB) 

BLANKS 

1 

52 

52 

001 

QUESTION  1 

1-6 

1 

53 

53 

002 

QUESTION  2 

1 

54 

54 

003 

QUEST10N3 

1 

55 

66 

004 

QUESTION  4 

1 

56 

56 

006 

QUESTION  5 

1 

57 

67 

Q06RES01 

QUESTION  6.  RESPONSE  1 

0.1 

1 

56 

68 

Q06RES02 

QUESTION  6,  RESPONSE  2 

0.1 

1 

59 

59 

Q06RES03 

QUESTION  6,  RESPONSE  3 

0.1 

1 

60 

60 

006RES04  QUESTION  6,  RESPONSE 4 

0.1 

1 

61 

61 

Q06RES05  QUESTION  6,  RESPONSE  5 

0.1 

1 

62 

62 

Q06RES06  OUESTX3N  6.  RESPONSE  6 

0,1 

1 

63 

63 

Q06RES07  QUESTION  6.  RESPONSE  7 

0.1 

1 

64 

64 

Q06RES06 

QUESTION  6.  RESPONSE  8 

0.1 

1 

65 

66 

006RES09 

QUESTION  6.  RESPONSE  9 

0.1 

1 

66 

66 

Q06RES10  QUESTION  6.  RESPONSE  10 

0.1 

1 

67 

67 

007 

QUESTION  7 

1-6 

1 

68 

68 

008 

QUESTIONS 

1-6 

1 

69 

69 

009 

QUESTIONS 

1-6 

1 

70 

70 

Q10 

QUESTION  10 

1-6 

1 

71 

71 

Oil 

QUESTION  11 

1-5 

1 

72 

72 

012 

QUESTION  12 

1-6 

1 

73 

73 

013 

QUESTION  13 

1-6 

1 

74 

74 

014 

QUESTION  14 

1-6 

1 

75 

75 

015 

QUESTION  15 

1-6 

1 

76 

76 

016 

QUESTION  16 

1-5 

1 

77 

77 

017 

QUESTION  17 

1-6 

1 

78 

78 

018 

QUESTION  18 

1-6 

1 

79 

79 

Q19 

QUESTION  19 

1-6 

1 

80 

80 

020 

QUESTION  20 

1-6 

1 

81 

81 

021 

QUESTION  21 

1-6 

1 

82 

82 

022 

QUESTION  22 

1-6 

1 

83 

83 

Q23 

QUESTION  23 

1-6 

1 

84 

84 

024 

QUESTION  24 

1-6 

1 

85 

85 

025 

QUESTION  25 

1-6 

1 

86 

86 

Q26 

QUESTION  26 

1 

87 

87 

027 

QUESTION  27 

1 

88 

88 

028 

QUESTION  2B 

1 

89 

89 

Q29RES01 

QUESTION  29,  RESPONSE  1 

0.1 

1 

90 

90 

Q29RES02  QUESTION 29.  RESPONSE 2 

0,1 

1 

91 

91 

Q29RES03 

QUESTION  29.  RESPONSE  3 

0.1 

1 

92 

92 

Q29RES04  QUESTION  29.  RESPONSE  4 

0,1 

1 

93 

93 

Q29RES05 

QUESTION  29.  RESPONSES 

0.1 

1 

94 

94 

Q29RES06 

QUESTION  29,  RESPONSE  6 

0.1 

1 

95 

95 

030 

QUESTION® 

1 

96 

96 

031 

QUESTION  31 

1 

97 

97 

032 

QUESTION® 

A-Z 

A-Z 


A-E 

A.B 

A-E 

A-G 


A-D 

A-C 

A-E 


A-E 

A-F 

A-C 
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FLD«  NC  SC  EC 


1  96  96 

1  99  99 

1  100  100 

1  101  101 

1  102  102 

1  103  103 

1  104  104 

1  106  106 

1  106  106 

1  107  107 

1  106  106 

1  109  109 

1  110  110 

1  111  111 

1  112  112 

1  113  113 

1  114  114 

1  116  116 

1  116  116 

1  117  117 

1  118  116 

1  119  1i» 

1  120  120 

1  121  121 

1  122  122 

1  123  123 

1  124  124 

1  125  125 

1  126  126 

1  127  127 

1  128  128 

1  129  129 

1  130  130 

1  131  131 

1  132  132 

1  133  133 

1  134  134 

1  135  135 

1  136  136 

1  137  137 

1  138  138 

1  139  139 

1  140  140 

1  141  141 

1  142  142 

1  143  143 

1  144  144 

1  145  145 

1  146  146 

1  147  147 

1  148  148 

1  149  149 

1  150  150 

1  151  151 

1  152  152 


name  description 


033  QUESTION  33 

034  QUESTION  34 

036  QUESTION  35 

Q36RES01  QUESTION  36,  RESPONSE  1 
Q36RES02  QUESTION  36.  RESPONSE  2 
Q36RES03  QUESTION  36.  RESPONSE  3 
Q36RES04  QUESTION  36,  RESPONSE  4 
Q36RES06  QUESTION 36,  RESPONSES 
Q36RES06  QUESTION  36.  RESPONSE  6 
Q36RES07  QUESTION  36.  RESPONSE  7 
Q36RES06  QUESTION 36.  RESPONSES 
Q36RES09  QUESTION  38.  RESPONSE  9 
Q37RES01  QUESTION  37.  RESPONSE  1 
Q37RES02  QUESTION  37.  RESPONSE  2 
Q37RES03  QUESTION  37.  RESPONSE  3 
Q37RES04  QUESTION  37.  RESPONSE  4 
Q37RES06  QUESTION  37.  RESPONSE  5 
Q37RES06  QUESTON  37,  RESPONSE  6 
Q37RES07  QUESTION  37.  RESPONSE  7 
Q37RES06  QUESTION 37.  RESPONSES 
Q37RES09  QUESTION  37,  RESPONSE  9 
Q38RES01  QUESTION  3&  RESPONSE  1 
Q38RES02  QUESTION  3ft  RESPONSE  2 
Q38RES03  QUESTION  SB.  RESPONSE  3 
Q38RES04  QUESTION  38,  RESPONSE  4 
Q38RES06  QUESTION  36.  RESPONSE  5 
Q38RES06  QUESTION  3a  RESPONSE  6 
Q38RES07  QUESTION  3a  RESP0NSE7 
Q38RES06  QUESTION  3a  RESPONSE  6 
Q38RES09  QUESTION 38.  RESPONSES 
Q39RES01  QUESTION  39,  RESPONSE  1 
Q39RES02  QUESTION  39.  RESPONSES 
Q39RES03  QUESTION  39.  RESPONSE  3 
Q39RES04  QUESTION  39.  RESPONSE  4 
Q39RES05  QUESTION 39,  RESPONSES 
Q39RES06  QUESTION  39.  RESPONSES 
Q39RES07  QUESTION  39,  RESPONSE  7 
Q£;  RES08  QUESTION  39,  RESPONSE  8 
Q39RES09  QUESTION  39.  RESPONSE  9 
Q40RES01  QUESTION  40,  RESPONSE  1 
Q40RES02  QUESTION  40,  RESPONSE 2 
Q40RES03  QUESTION 4a  RESPONSES 
Q40RES04  QUESTION  40,  RESPONSE  4 
Q40RES05  QUESTION  40,  RESPONSE  5 
Q40RES06  QUESTION  4a  RESPONSE  6 
040RES07  QUESTION  40,  RESPONSE  7 
Q40RES06  QUESTION  40,  RESPONSE  8 
Q40RES09  QUESTION  4a  RESPONSE  9 
Q41RES01  QUESTION  41.  RESPONSE  1 
Q41RES02  QUESTION 41  .RESPONSE 2 
Q41*'ES03  QUESTION 41.  RESPONSES 
Q41HES04  QUESTION  41.  RESPONSE  4 
Q41RES06  QUEST10N41,  RESPONSES 
Q41RES06  QUESTION 41,  RESPONSES 
Q41RES07  QUESTION  41.  RESPONSE  7 


RANGE 


1-6 

1-6 

1-6 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0,1 

0.1 

0.1 

ai 

0,1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0,1 

0,1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0,1 

0.1 

o.l 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0.1 

0,1 
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FLO*  NC 

SC 

EC 

NAME  DESCRIPTION 

RANGE 

100 

1 

153 

153 

O41RES08  QUESTION 41  .RESPONSE 8 

0.1 

110 

1 

154 

154 

Q41RES09  QUESTION  41,  RESPONSE  9 

0.1 

111 

1 

165 

155 

Q42RES01  QUESTION  42,  RESPONSE  1 

0,1 

112 

1 

156 

166 

Q42RES02  QUESTION  42,  RESPONSE  2 

0.1 

113 

1 

157 

157 

Q42RES03  QUESTION  42.  RESPONSE  3 

0.1 

114 

1 

158 

158 

Q42RES04  QUESTION  42.  RESPONSE  4 

0.1 

lift 

1 

150 

159 

Q42RES05  QUESTION  42,  RESPONSE  5 

0.1 

116 

1 

180 

180 

O42RES06  QUESTION  42,  RESPONSE  6 

0.1 

117 

1 

161 

161 

Q42RES07  QUESTION  42,  RESPONSE  7 

0.1 

118 

1 

162 

162 

Q42RES08  QUESTION  42.  RESPONSE  8 

0.1 

119 

1 

163 

163 

Q42RES09  QUESTION  42,  RESPONSE  0 

0.1 

120 

1 

164 

164 

Q43RES01  QUESTION  43,  RESPONSE  1 

0.1 

121 

1 

165 

166 

Q43RES02  QUESTION  43,  RESPONSE  2 

0.1 

122 

1 

106 

166 

Q43RES03  OUESTTON  43.  RESPONSE  3 

0.1 

123 

1 

167 

167 

Q43RES04  QUESTION  43,  RESPONSE  4 

0.1 

124 

1 

168 

166 

Q43RES05  QUESTION  43.  RESPONSE  5 

0.1 

125 

1 

160 

160 

Q43RES06  QUESTION  43,  RESPONSE  6 

0.1 

125 

1 

170 

170 

Q43RES07  QUESTION  43.  RESPONSE  7 

0.1 

127 

1 

171 

171 

Q43RES06  QUESTION  43,  RESPONSE  8 

0.1 

128 

1 

172 

172 

Q43RES09  QUESTION  43,  RESPONSE  9 

0.1 

120 

1 

173 

173 

Q44RES01  QUESTION  44,  RESPONSE  1 

0,1 

130 

1 

174 

174 

Q44RES02  QUESTION  44,  RESPONSE  2 

0.1 

131 

1 

175 

175 

Q44RES03  QUESTION  44.  RESPONSE  3 

0.1 

132 

1 

176 

176 

Q44RES04  QUESTION  44,  RESPONSE  4 

0.1 

133 

1 

177 

177 

Q44RES06  QUESTION  44.  RESPONSE  5 

0.1 

134 

1 

178 

178 

Q44RES06  QUESTION  44,  RESPONSE  6 

0.1 

135 

1 

179 

170 

O44RES07  QUESTION  44,  RESPONSE  7 

0.1 

136 

1 

180 

180 

Q44RES06  QUESTION  44,  RESPONSE  8 

0.1 

137 

1 

181 

181 

O44RESO0  QUESTION  44,  RESPONSE  9 

0.1 

136 

1 

182 

182 

Q45RES01  QUESTION  45,  RESPONSE  1 

0,1 

139 

1 

183 

183 

Q45RES02  QUESTION  45.  RESPONSE  2 

0,1 

140 

1 

184 

184 

Q45RES03  QUESTION 45,  RESPONSES 

0.1 

141 

1 

185 

185 

045RES04  QUESTION  45,  RESPONSE  4 

0.1 

142 

1 

186 

186 

Q45RES05  QUESTION 45.  RESPONSES 

0.1 

143 

1 

187 

187 

Q45RES06  QUESTION  45,  RESPONSE  6 

0.1 

144 

1 

188 

188 

Q45RES07  QUESTION  45,  RESPONSE  7 

0.1 

145 

1 

189 

180 

Q45RES08  QUESTION  45.  RESPONSE  8 

0.1 

146 

1 

190 

100 

Q45RES09  QUESTION  45,  RESPONSE  9 

0.1 

147 

1 

191 

191 

Q46RES01  QUESTION  46,  RESPONSE  1 

0.1 

148 

1 

102 

192 

Q46RES02  QUESTION  45,  RESPONSE  2 

0.1 

140 

1 

193 

193 

Q46RES03  QUESTION  46,  RESPONSE  3 

0,1 

150 

1 

104 

194 

046RES04  QUESTION  46,  RESPONSE  4 

0,1 

151 

1 

105 

195 

Q46RES06  QUESTION  46,  RESPONSE  5 

0,1 

152 

1 

196 

196 

Q46RES06  QUESTION  46,  RESPONSE  6 

0.1 

153 

1 

197 

197 

Q46RES07  QUESTION  46,  RESPONSE  7 

0,1 

154 

1 

108 

198 

Q46RES06  QUESTION  46,  RESPONSE  8 

0.1 

155 

1 

109 

199 

Q46RES09  QUESTION  46,  RESPONSE  9 

0,1 

156 

1 

200 

200 

Q47RES01  QUESTION  47,  RESPONSE  1 

0.1 

157 

1 

201 

201 

Q47RES02  QUESTION  47,  RESPONSE  2 

0.1 

158 

1 

202 

202 

Q47RES03  QUESTION  47,  RESPONSE  3 

0,1 

150 

1 

203 

203 

Q47RES04  QUESTION  47,  RESPONSE  4 

0,1 

160 

1 

204 

204 

Q47RES06  QUESTION 47,  RESPONSES 

0,1 

161 

1 

206 

205 

Q47RES06  QUESTION  47,  RESPONSE  6 

0.1 

162 

1 

206 

206 

Q47RES07  QUESTION  47.  RESPONSE  7 

0,1 

163 

1 

207 

207 

Q47RES06  QUESTION  47,  RESPONSE  8 

0,1 
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FUDf  NC 

SC 

EC 

NAME 

DESCRIPTION  RANGE 

164 

1 

206 

206 

Q47RE509 

QUESTION  47,  RESPONSES 

0.1 

16S 

1 

209 

209 

048 

QUESTION  48 

A-E 

166 

1 

210 

210 

049 

QUESTION  49 

A-C 

167 

1 

211 

211 

Q50RES01 

QUESTION  SO,  RESPONSE  1 

0,1 

168 

1 

212 

212 

Q50RESQ2 

QUESTION  50,  RESPONSE  2 

0.1 

168 

1 

213 

213 

Q50RES03 

QUESTION  50,  RESPONSE  3 

0,1 

170 

1 

214 

214 

Q50RES04 

QUESTION  50,  RESPONSE  4 

0,1 

171 

1 

215 

215 

Q50RES06 

QUESTION  50.  RESPONSE  5 

0.1 

172 

1 

216 

216 

Q50RES06 

QUESTION  60,  RESPONSES 

0.1 

173 

1 

217 

217 

Q50RES07 

QUESTION  5ft  RESPONSE  7 

0.1 

174 

1 

218 

218 

051 

QUESTION  51 

1«6 

175 

1 

219 

219 

052 

QUESTION  52 

14 

178 

2 

220 

221 

MAJCOM 

ASGT -CURRSJT*  MAJOR  OOMMANDD 

A-Z 

177 

6 

222 

227 

PAFSC 

AFSC  •  PRMARV  •  NUMBERiSUFFIX 

0-9, 

AC 

178 

6 

228 

233 

DAF5C 

APSC-DUTY-NUMBEFVSUmX 

0-9. 

A-C 

178 

17 

234 

250 

DnrvLOc 

ASGT  -  DUTY  LOCATION  NAME 

A-Z 

180 

2 

251 

252 

GRADE 

CURRENT  GRADE 

04 
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STRUCTURE  OF  MANAGER  DATA 


FLD* 

Field  Name 

Type 

NC 

1 

RECJMO 

Numeric 

3 

2 

CE111 

Numeric 

1 

3 

MAINT112 

Numeric 

1 

4 

MED113 

Numeric 

1 

5 

MS114 

Numeric 

1 

6 

PERS115 

Numeric 

1 

7 

SP116 

Numeric 

1 

8 

SUP117 

Numeric 

1 

9 

TRANS118 

Numeric 

1 

10 

MAC121 

Numeric 

1 

11 

SAC122 

Numeric 

1 

12 

TAC123 

Numeric 

1 

13 

AFSC13 

Character 

7 

14 

PAFSC131 

Character 

7 

15 

75XXX132 

Numeric 

4 

16 

TTS14 

Numeric 

4 

17 

FRU61 

Numeric 

1 

18 

ADD162 

Numeric 

1 

19 

MEASTM17 

Numeric 

4 

20 

N0181 

Numeric 

1 

21 

YES182 

Numeric 

1 

22 

N0191 

Numeric 

1 

23 

YES192 

Numeric 

1 

24 

LC1921 

Numeric 

1 

25 

NRH1922 

Numeric 

1 

26 

DIM1923 

Numeric 

1 

27 

ENH1924 

Numeric 

1 

28 

FLEX1925 

Numeric 

1 

29 

CMS1101 

Numeric 

1 

30 

SMS1102 

Numeric 

1 

31 

MSG1103 

Numeric 

1 

32 

TSG1104 

Numeric 

1 

33 

SSG1105 

Numeric 

1 

34 

SGT1106 

Numeric 

1 

35 

RAMN1107 

Numeric 

1 

36 

A1C1108 

Numeric 

1 

37 

AMN1109 

Numeric 

1 

38 

NBA11010 

Numeric 

1 
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FID# 


Field  Name 


NC 


OV11011 

POPE1111 

LANG1112 

OFnil3 

comm 

CCON1115 

LUS1116 

N0211 

YES212 

TV221 

EM222 

OY223 

SM224 

TM225 

LTTM2226 

FORE231 

MEC232 

TTP233 

FTD234 

COMP235 

EVALS236 

INC241 

DEC242 

SAME243 

ENTRY251 

EQUIP252 

NCMAN253 

ECMAN254 

DECTR255 

DMCTR256 

NCMIS257 

ECM3S258 

STDS259 

EMPH2510 

NSOL2511 

YES261 

CTR26111 


Type 


Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  I 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 

Numeric  1 


IMP26112  Numeric  1 

N0262  Numeric  1 

CTR26211  Numeric  1 


IMP26212  Numeric  1 

N0271  Numeric  1 
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ELD# 

Held  Name 

Type 

NC 

81 

YES272 

Numeric 

1 

82 

ASS27211 

Numeric 

1 

83 

OJT27212 

Numeric 

1 

84 

NTR27213 

Numeric 

1 

85 

N0311 

Numeric 

1 

86 

YES312 

Numeric 

1 

87 

MAN321 

Numeric 

1 

88 

AUT0322 

Numeric 

1 

89 

IMS32211 

Numeric 

1 

90 

AMS32212 

Numeric 

1 

91 

PAS32213 

Numeric 

1 

92 

ANG32214 

Numeric 

1 

93 

CAL32221 

Numeric 

1 

94 

EED32231 

Numeric 

1 

95 

CUR32232 

Numeric 

1 

96 

ANG32233 

Numeric 

1 

97 

EFF32234 

Numeric 

1 

98 

ONE32236 

Numeric 

1 

99 

OMP32241 

Numeric 

1 

100 

UMB32242 

Numeric 

1 

101 

06132243 

Numeric 

1 

102 

FLX32244 

Numeric 

1 

103 

ONE32246 

Numeric 

1 

104 

UMD331 

Numeric 

1 

105 

ATT332 

Numeric 

1 

106 

LMC333 

Numeric 

1 

107 

LCST334 

Numeric 

1 

108 

LASS335 

Numeric 

1 

109 

SHOWS336 

Numeric 

1 

110 

ETA337 

Numeric 

1 

111 

GTTG338 

Numeric 

1 

112 

FTDAV339 

Numeric 

1 

113 

NFSC3310 

Numeric 

1 

114 

NONE3312 

Numeric 

1 

115 

REC.NO 

Numeric 

3 

116 

CBT411 

Numeric 

1 

117 

IVD412 

Numeric 

1 

118 

VIDE0413 

Numeric 

1 

119 

CLSRM414 

Numeric 

1 

120 

SELF415 

Numeric 

1 

121 

SIM416 

Numeric 

1 

122 

EQUEP417 

Numeric 

1 
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Field  Name 


Type 


NOFEQ418 

AVMON421 

VTTME422 

SSINS423 

EDBCK424 

NDSON425 

ISRUP426 

REAL427 

NONE429 

COST431 

WASTE432 

REINS433 

OFEED434 

HNDON435 

1SRUP436 

OREAL437 

TDATE438 

OTIME439 

RING4310 

NONE4312 

N0441 

YES442 

RBBM451 

XPENS452 

WHME453 

EQOBS454 

N0511 

YES512 

FIFTY521 

TFTF522 

TTTF523 

LTEN524 

N0531 

YES532 

ISD541 

AUT0542 

NA551 

OTFAM552 

MPLEX553 

UMBER554 

NA555 

MECON556 


Numeric 

Character 

Character 

Character 

Character 

Character 

Character 

Character 

Numeric 

Character 

Character 

Character 

Character 

Character 

Character 

Character 

Character 

Character 

Character 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 


FLD* 

Field  Name 

Type 

NC 

165 

OSTLY557 

Numeric 

1 

166 

REGS561 

Numeric 

1 

167 

CDC562 

Numeric 

1 

168 

ISD563 

Numeric 

1 

169 

SME564 

Numeric 

1 

170 

REGS571 

Numeric 

1 

171 

CDC572 

Numeric 

1 

172 

ISD573 

Numeric 

1 

173 

SME574 

Numeric 

1 

174 

BTS575 

Numeric 

1 

175 

CONT576 

Numeric 

1 

176 

FTD577 

Numeric 

1 

177 

N0611 

Numeric 

1 

178 

YES612 

Numeric 

1 

179 

MAN6121 

Numeric 

1 

180 

AUT06122 

Numeric 

1 

181 

CAL61221 

Numeric 

1 

182 

PAPER621 

Numeric 

1 

183 

73ME622 

Numeric 

1 

184 

UMBER623 

Numeric 

1 

185 

STAND624 

Numeric 

1 

186 

VALUE625 

Numeric 

1 

187 

AUT0626 

Numeric 

1 

188 

RRENT627 

Numeric 

1 

189 

SGU1D628 

Numeric 

1 

190 

NONE6210 

Numeric 

1 

191 

N0711 

Numeric 

1 

192 

YES712 

Numeric 

1 

193 

OTS721 

Numeric 

1 

194 

QUEST722 

Numeric 

1 

195 

NTSUP723 

Numeric 

1 

196 

EXAMS724 

Numeric 

1 

197 

SKEVA725 

Numeric 

1 

198 

SME726 

Numeric 

1 

199 

DNK728 

Numeric 

1 

200 

N0731 

Numeric 

1 

201 

YES732 

Numeric 

1 

202 

CBT73211 

Numeric 

1 

203 

CAL73221 

Numeric 

1 

204 

NOSH 

Numeric 

1 

205 

YES812 

Numeric 

1 

206 

NPORM821 

Numeric 

1 
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ELD# 

Field  Name 

Type 

NC 

207 

RMALA822 

Numeric 

1 

208 

RMALS823 

Numeric 

1 

209 

IANN8231 

Numeric 

1 

210 

UART8232 

Numeric 

1 

211 

ONTH8233 

Numeric 

1 

212 

WEEK8234 

Numeric 

1 

213 

AILY8235 

Numeric 

1 

214 

NOS31 

Numeric 

1 

215 

YES832 

Numeric 

1 

216 

ARE83211 

Numeric 

1 

217 

ARN83212 

Numeric 

1 

218 

LVE83213 

Numeric 

1 

219 

ORD83214 

Numeric 

1 

220 

INFE3215 

Numeric 

1 

221 

RBC.NO 

Numeric 

3 

222 

YES911 

Numeric 

1 

223 

EP9111 

Numeric 

1 

224 

JFTY9112 

Numeric 

1 

225 

TT9113 

Numeric 

1 

226 

IRTY9114 

Numeric 

1 

227 

F9115 

Numeric 

1 

228 

ENTY9116 

Numeric 

1 

229 

FTEN9117 

Numeric 

1 

230 

TEN9118 

Numeric 

1 

231 

UNK9119 

Numeric 

1 

232 

N0912 

Numeric 

1 

233 

ISSUF921 

Numeric 

1 

234 

MANPR922 

Numeric 

1 

235 

QUAL923 

Numeric 

1 

236 

WAY5B924 

Numeric 

1 

237 

EWAYT925 

Numeric 

1 

238 

AUMP926 

Numeric 

1 

239 

AILOR927 

Numeric 

1 

240 

OFAKT928 

Numeric 

1 

241 

DNK1031 

Numeric 

1 

242 

YES1032 

Numeric 

1 

243 

CBT10321 

Numeric 

1 

244 

IVD10322 

Numeric 

1 

245 

NO1041 

Numeric 

1 

246 

YES1111 

Numeric 

1 

247 

N01112 

Numeric 

1 
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FLD# 

Field  Name 

Type 

NC 

87 

SGT1196 

Numeric 

1 

88 

RAMN1197 

Numeric 

1 

89 

A1C1198 

Numeric 

1 

90 

AMN1199 

Numeric 

1 

91 

NBA11910 

Numeric 

1 

92 

OV11911 

Numeric 

1 

93 

POFE1201 

Numeric 

1 

94 

LANG1202 

Numeric 

1 

95 

OFF1203 

Numeric 

1 

96 

COTT1204 

Numeric 

1 

97 

CCON1205 

Numeric 

1 

98 

LUS1206 

Numeric 

1 

99 

REC_NO 

Numeric 

3 

100 

N0211 

Numeric 

1 

101 

YES212 

Numeric 

1 

102 

SDCM0221 

Numeric 

i 

103 

QNEYR222 

Numeric 

1 

104 

EM0223 

Numeric 

1 

105 

TWOYR224 

Numeric 

1 

106 

NQ231 

Numeric 

1 

107 

YES232 

Numeric 

1 

108 

UTO23211 

Numeric 

1 

109 

MAN23212 

Numeric 

1 

110 

U 123221 

Numeric 

1 

111 

NGN23222 

Numeric 

1 

112 

UIP23223 

Numeric 

1 

113 

ONT23231 

Numeric 

1 

114 

NTH23232 

Numeric 

1 

115 

ART23233 

Numeric 

1 

116 

ANN23234 

Numeric 

1 

117 

UAL23235 

Numeric 

1 

118 

SND23236 

Numeric 

1 

119 

N023241 

Numeric 

1 

120 

YES23242 

Numeric 

1 

121 

TEC23251 

Numeric 

1 

122 

TDC23252 

Numeric 

1 

123 

N0311 

Numeric 

1 

124 

YES312 

Numeric 

1 

125 

YEAR321 

Numeric 

1 

126 

EMOS322 

Numeric 

1 

127 

SMOS323 

Numeric 

1 

128 

FMOS324 

Numeric 

1 

129 

THMOS325 

Numeric 

1 

130 

WOMOS326 

Numeric 

1 

131 

ONEM0327 

Numeric 

1 
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FLD# 

Fieid  Name 

Type 

NC 

132 

UNK328 

Numeric 

1 

133 

MFR331 

Numeric 

1 

134 

TYSCH332 

Numeric 

1 

135 

INC341 

Numeric 

1 

136 

DEC342 

Numeric 

1 

137 

SAME343 

Numeric 

1 

138 

TFERS351 

Numeric 

1 

139 

BQUIP352 

Numeric 

1 

140 

NCMAN353 

Numeric 

1 

141 

ECMAN354 

Numeric 

1 

142 

DECTR355 

Numeric 

1 

143 

INCTR356 

Numeric 

1 

144 

CMISS357 

Numeric 

1 

145 

CMI5S358 

Numeric 

1 

146 

VSTDS359 

Numeric 

1 

147 

DEMP3510 

Numeric 

1 

148 

SMER3511 

Numeric 

1 

149 

LCH3512 

Numeric 

1 

150 

YE5361 

Numeric 

1 

151 

CTR36111 

Numeric 

1 

152 

ONE36112 

Numeric 

1 

153 

N0362 

Numeric 

1 

154 

CTR36211 

Numeric 

1 

155 

TN036212 

Numeric 

1 

156 

N0371 

Numeric 

1 

157 

YES372 

Numeric 

1 

158 

REF37211 

Numeric 

1 

159 

ELF37212 

Numeric 

1 

160 

N0411 

Numeric 

1 

161 

YES412 

Numeric 

1 

162 

MAN421 

Numeric 

1 

163 

AUT0422 

Numeric 

1 

164 

CAL42211 

Numeric 

1 

165 

EED42221 

Numeric 

1 

166 

ACC42222 

Numeric 

1 

167 

ANG42223 

Numeric 

1 

168 

EFF42224 

Numeric 

1 

169 

NE42226 

Numeric 

1 

170 

MP42231 

Numeric 

1 

171 

MB42232 

Numeric 

1 

172 

ST42233 

Numeric 

1 

173 

LX42234 

Numeric 

1 

174 

NE42236 

Numeric 

1 

175 

UMD431 

Numeric 

1 
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FLD# 

Field  Name 

Type 

NC 

176 

ATT432 

Numeric 

1 

177 

LMC433 

Numeric 

1 

178 

LCS434 

Numeric 

1 

179 

LASS435 

Numeric 

1 

180 

TYINT436 

Numeric 

1 

181 

OTXME437 

Ntuneric 

1 

182 

NONE439 

Numeric 

1 

183 

REC.NO 

Numeric 

3 

184 

CBT511 

Numeric 

1 

185 

IVD512 

Numeric 

1 

186 

VIDE0513 

Numeric 

1 

187 

CLSRM514 

Numeric 

1 

188 

SELF515 

Numeric 

1 

189 

SIM516 

Numeric 

1 

190 

EQUIP517 

Numeric 

1 

m 

NOPEQ518 

Numeric 

1 

192 

N0541 

Numeric 

1 

193 

YES542 

Numeric 

1 

194 

RBBM551 

Numeric 

1 

195 

XPENS552 

Numeric 

1 

196 

WTIME553 

Numeric 

1 

197 

EQOBS554 

Numeric 

1 

198 

N0611 

Numeric 

1 

199 

YES612 

Numeric 

1 

200 

FIFTY621 

Numeric 

1 

201 

TFTF622 

Numeric 

1 

202 

TTTF623 

Numeric 

1 

203 

LTEN624 

Numeric 

1 

204 

N0631 

Numeric 

1 

205 

VES632 

Numeric 

1 

206 

ISD641 

Numeric 

1 

207 

NA651 

Numeric 

1 

208 

OTFAM652 

Numeric 

1 

209 

MPLEX653 

Numeric 

1 

210 

UMBER654 

Numeric 

1 

211 

NA655 

Numeric 

1 

212 

MECON656 

Numeric 

1 

213 

OSTLY657 

Numeric 

1 

214 

REGS661 

Numeric 

1 

215 

CDC662 

Numeric 

1 

216 

STS663 

Numeric 

1 

217 

TM671 

Numeric 

1 

218 

T5EC672 

Numeric 

1 

219 

USAJP673 

Numeric 

1 
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FLO# 

Field  Name 

220 

REGS674 

221 

N0711 

222 

YES712 

223 

MAN721 

224 

AUTOT22 

225 

NAME7221 

226 

CAL72221 

227 

PAFER731 

228 

TEME732 

229 

UMBER733 

230 

STAND734 

231 

VALUE735 

232 

AUT0736 

233 

ATTME737 

234 

AONF738 

235 

NONE731J 

236 

NOGll 

237 

YES812 

238 

OTS821 

239 

QUEST822 

240 

NTSUP823 

241 

EXAMS824 

242 

N0831 

243 

YES832 

244 

CAL83221 

245 

R EC  NO 

246 

CONST911 

247 

DAILY912 

248 

OWEEK913 

249 

EEKLY914 

250 

NTHLY915 

251 

ARTLY916 

252 

NFORM921 

253 

I MALA922 

254 

RMALS923 

255 

ILY92311 

256 

EEK92312 

257 

NTH 923 13 

258 

ART92314 

259 

N0931 

260 

WHY9311 

261 

YES932 

262 

ARE93211 

263 

ARN93212 

264 

LVE93213 

Type 


NC 


Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 


1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

3 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 
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FLD# 

Field  Name 

Type 

NC 

265 

ORD93214 

Numeric 

1 

266 

ROB93215 

Niuneric 

1 

267 

ELP93216 

Numeric 

1 

268 

YES1011 

Numeric 

1 

269 

UND10111 

Numeric 

1 

270 

FIY10112 

Numeric 

1 

271 

0010113 

Numeric 

1 

272 

IVE10114 

Numeric 

1 

273 

ENT10115 

Numeric 

1 

274 

EN10116 

Numeric 

1 

275 

TEN10117 

Numeric 

1 

276 

DK10118 

Numeric 

1 

277 

NO1012 

Numeric 

1 

278 

SSUF1021 

Numeric 

1 

279 

ANPR1022 

Numeric 

1 

280 

QUAL1023 

Numeric 

1 

281 

WAYB1Q24 

Numeric 

1 

282 

WAYT1025 

Numeric 

1 

283 

AUM1026 

Numeric 

1 

284 

ILOR1027 

Numeric 

1 

285 

OART1028 

Numeric 

1 

286 

DSON1111 

Niuneric 

1 

2 87 

EXER1112 

Niuneric 

1 

288 

TEME1121 

Niuneric 

1 

289 

MPR1122 

Numeric 

1 

290 

QUIP1123 

Niuneric 

1 

291 

DATE1124 

Numeric 

1 

292 

NONE112 6 

Numeric 

1 

293 

DK1131 

Niuneric 

1 

294 

YES1132 

Numeric 

1 

295 

CBT11321 

Numeric 

1 

296 

IVD11322 

Numeric 

1 

297 

DEOH323 

Numeric 

1 

298 

SIM11324 

Numeric 

1 

299 

NOH41 

Numeric 

1 

300 

YES1142 

Numeric 

1 

301 

IME11421 

Numeric 

1 

302 

NDS11422 

Numeric 

1 

303 

RTM11423 

Numeric 

1 

304 

YES1211 

Numeric 

1 

305 

N01212 

Numeric 

1 
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STRUCTURE  OF  TRAINEE  DATA 


FLD# 

Field  Name 

Type 

NC 

1 

REC.NO 

Numeric 

3 

2 

CE111 

Numeric 

1 

3 

MAINT112 

Numeric 

1 

4 

MED113 

Numeric 

1 

5 

MSI  14 

Numeric 

1 

6 

FERS115 

Numeric 

1 

7 

SP116 

Numeric 

1 

8 

SUP117 

Numeric 

1 

9 

TRANS118 

Numeric 

1 

10 

MAC121 

Numeric 

1 

11 

SAC122 

Numeric 

1 

12 

TAC123 

Numeric 

1 

13 

AFSC13 

Character 

7 

14 

T1S14 

Numeric 

4 

15 

YES151 

Numeric 

1 

16 

OJT1511 

Numeric 

1 

17 

FTD1512 

Numeric 

1 

18 

CDC1513 

Numeric 

1 

19 

NQL1514 

Numeric 

1 

20 

PME1515 

Numeric 

1 

21 

N0152 

Numeric 

1 

22 

OJT1521 

Numeric 

1 

23 

FTD1522 

Numeric 

1 

24 

CDC1523 

Numeric 

1 

25 

NQL1524 

Numeric 

1 

26 

PME1525 

Numeric 

1 

27 

NONE1527 

Numeric 

1 

28 

H161 

Numeric 

1 

29 

ENN162 

Numeric 

1 

30 

SSN163 

Numeric 

1 

31 

FFN164 

Numeric 

1 

32 

TTN165 

Numeric 

1 

33 

LTT166 

Numeric 

1 

34 

CMS171 

Numeric 

1 

35 

SMS172 

Numeric 

1 

36 

MSG173 

Numeric 

1 

37 

TSG174 

Numeric 

1 

38 

SSG175 

Numeric 

1 
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83a^Sia2a3aaS8558!8!£eSSS!88!!l*aXBaa86fe<5*;ftt6fiSg!S 


ELD# 


Held  Name 

Type 

NC 

SGT176 

Numeric 

1 

SRAMN177 

Numeric 

1 

A1C178 

Numeric 

1 

AMN179 

Numeric 

1 

NBAS1710 

Numeric 

1 

OV1711 

Numeric 

1 

FOFE181 

Numeric 

1 

LANG182 

Numeric 

1 

OFF183 

Numeric 

1 

SCOm  84 

Numeric 

1 

MCCON185 

Numeric 

1 

ELLZS186 

Numeric 

1 

0TECH211 

Numeric 

1 

OUNTT212 

Numeric 

1 

ITBET213 

Numeric 

1 

CHBET214 

Numeric 

1 

SAME215 

Numeric 

1 

OOPTN216 

Numeric 

1 

NTECH221 

Numeric 

1 

TZME2211 

Numeric 

1 

VXRN2212 

Numeric 

1 

U1F2213 

Numeric 

1 

QUAL2214 

Numeric 

1 

HME2215 

Numeric 

1 

NSIS2216 

Numeric 

1 

ORGN2217 

Numeric 

1 

ROAD2218 

Numeric 

1 

NUN1T222 

Numeric 

1 

REAL2221 

Numeric 

1 

DSON2222 

Numeric 

1 

ILOR2223 

Numeric 

1 

ALTR2224 

Numeric 

1 

NEON2225 

Numeric 

1 

EQUAL223 

Numeric 

1 

CBT311 

Numeric 

1 

IVD312 

Numeric 

1 

VIDE0313 

Numeric 

1 

CLSRM314 

Numeric 

1 

SELF315 

Numeric 

1 

SIM316 

Numeric 

1 

EQUIP317 

Numeric 

1 

NONOP318 

Numeric 

1 
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|*S!3*iS!ea8S8888SJSffi88SBSS 


Held  Name 


Type 


DK3110 

CBT321 

IVD322 

VIDE0323 

CLSRM324 

SELF325 

SIM326 

EQUIP327 

NONOP328 

NONE3210 

ANDON331 

EDBCK332 

ETRHF333 

NPACE334 

SELF335 

CBT341 

IVD342 

VEDE0343 

CLSRM344 

SELF345 

SZM346 

EQUIP347 

NONOP348 

NONE3410 

ORJNG351 

OHAND352 

NREAL353 

REC.NO 

OBS411 

QUEST412 

PEINT413 

EXAM414 

YES421 

N0422 

DK423 

YES431 

N0432 

FER43211 

YES511 

N0512 


Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 
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STRUCTURE  OF  DESERT  STORM  DATA 


FLDt 

Held  Name 

Type 

NC 

1 

REG  NO 

Numeric 

3 

2 

NOlll 

Numeric 

1 

3 

YES112 

Numeric 

1 

4 

DEAST121 

Numeric 

1 

5 

ARECH122 

Numeric 

1 

6 

CONUS123 

Numeric 

1 

7 

OTHER124 

Character 

25 

8 

VEGHT131 

Numeric 

1 

9 

VESDC132 

Numeric 

1 

10 

REFOR133 

Numeric 

1 

11 

NETW0134 

Numeric 

1 

12 

R1DUTY14 

Character 

50 

13 

TM151 

Numeric 

1 

14 

TR152 

Numeric 

1 

15 

TE153 

Numeric 

1 

16 

COMDR154 

Numeric 

1 

17 

SUPER155 

Numeric 

1 

18 

NONE156 

Numeric 

1 

19 

FREP211 

Numeric 

1 

20 

OFREF212 

Numeric 

1 

21 

YNOT2121 

Character 

100 

22 

RMOJT221 

Numeric 

1 

23 

EWSKL222 

Numeric 

1 

24 

RHGN223 

Numeric 

1 

25 

ANOL224 

Numeric 

1 

26 

EOAL225 

Numeric 

1 

27 

OTHER226 

Character 

100 

28 

NONE227 

Numeric 

1 

29 

IMMED231 

Numeric 

1 

30 

INDEL232 

Numeric 

1 

31 

NSDEL233 

Numeric 

1 

32 

OLATE234 

Numeric 

1 

33 

SUPER241 

Numeric 

1 

34 

TR242 

Numeric 

1 

35 

TDDMS243 

Numeric 

1 

36 

ME244 

Numeric 

1 

37 

PERTS245 

Numeric 

1 

38 

OWORK246 

Numeric 

1 

39 

OTHER247 

Character 

100 
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FLD# 


ndd  Name  TyP« 


NC 


SREQS251 

AUNS252 

ATTJRQ253 

GFACS254 


NONE257 

2NSUK261 

INSUK262 

TEAM2621 

THER2622 

C0NTZ71 

AIVED272 

OTREQ273 

NDSON311 

SELF312 

GROUP313 

CAD14 

IVD315 

VQR316 

SIM317 

CTOSE316 

OXHER319 

NDSON321 

SELF322 

GR0UP323 

CAD24 

IVD325 

VCR326 

SIM327 

CTUKE328 

OTHER329 

NDSON331 

OREAL332 

ONLY333 

OTHER334 

NDSON341 

SELF342 

GROUP343 

CAD44 

IVD345 

VCR346 


Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Character 

Numeric 

Numeric 

Numeric 

Numeric 

Character 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Character 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Character 

Numeric 

Numeric 

Numeric 

Character 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 

Numeric 


1 

1 

1 

1 

1 

50 

1 

1 

1 

1 

100 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

25 

1 

1 

1 

1 

1 

1 

1 

1 

25 

1 

1 

1 

150 

1 

1 

1 

1 

1 

1 
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FLD# 


Fiel4  Name 

Type 

NC 

SIM347 

Numeric 

1 

CTURE348 

Numeric 

1 

OTHER349 

Character 

50 

NONE3410 

Numeric 

1 

YNOWRK35 

Character 

100 

N0361 

Numeric 

1 

YES362 

Numeric 

1 

DSON3621 

Numeric 

1 

SELF3622 

Numeric 

1 

ROUP3623 

Numeric 

1 

CAD624 

Numeric 

1 

IVD3625 

Numeric 

1 

VCR3626 

Numeric 

1 

SIM3627 

Numeric 

1 

CTUR3628 

Numeric 

1 

THER3629 

Character 

100 
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DATA  PROCESSING  AND  DATA  ANALYSIS  PLAN 


This  analysis  plan  assumes  that  data  will  be  entered  into  a  computer  readable  ftle  by  the  Opncal 
Scanning  procedure. 

1 .  Lay-out  of  Analytic  Data  Files.  The  creation  of  analytic  data  files  from  the  OpScan  form 
should  be  done  by  a  person  experienced  with  the  data  entry  formats  and  procedures  to  be  used 
for  analysis.  Data  fields  will  be  of  fixed  length,  from  the  OpScan  input.  For  items  that  are  of 
the  "Mark  all  that  apply"  type,  a  separate  data  field  should  be  used  for  each  alternative,  entering 
a  "1”  if  checked,  and  "0"  if  not  checked.  This  permits  easy  analysis  and  easily  created  summary 
variables,  such  as  the  number  of  items  checked  on  the  question.  For  items  with  a  response 
scale,  e.g.,  "strongly  agree"  to  “'.trongly  disagree",  the  response  lencrs  should  be  convened  to  a 
numeric  1  to  5  scale,  to  permit  quantitative  analysis. 

Explicit  conventions  should  be  set  up  for  handling  missing  data  elements  and  the  "don’t  know/not 
applicable"  responses  -  these  conventions  vary  with  the  software  being  used  for  analysis,  for 
example  SAS  or  SPSS.  "Missing  data"  also  contain  information  and  needs  to  be  handled 
carefully,  particularly  so  that  these  data  elements  do  not  interfere  with  the  analysis  of  scaled  data. 
For  example,  if  ”9”  is  used  as  a  code  for  a  "Not  Applicable/Don't  Know"  response  (a  common 
convention),  the  software  should  be  coded  to  exclude  "9"  from  quantitative  calculations. 

2.  Analysis  Software.  Use  a  software  package  designed  for  survey-type  data  analysis,  such  as 
SPSS  or  SAS.  Both  SAS  and  SPSS  have  advantages  and  disadvantages.  SPSS's  procedures 
for  showing  frequencies  and  simple  tables  work  better  than  those  in  SAS  Also,  SPSS  has  a 
very  convenient  procedure  (called  BREAKDOWN)  for  examining  the  mean  responses  to  a 
scaled  item  for  different  subgroups,  such  as  those  from  different  AFSCs.  gTade  levels,  or 
lengths  of  service.  SAS  has  nothing  comparable,  although  tt  is  possible  to  obtain  the  same 
results.  A  key  criterion  in  choosing  the  software  is  the  familiarity  of  the  analyst  with  the 
software  package  to  be  used. 

3  Data  Cleaning.  After  data  entry,  as  the  first  task  after  creating  each  analytical  fiie.  numerical 
data  should  be  examined  for  any  illegal  values,  or  extreme  outliers.  If  outliers  are  found,  check 
against  other  data  elements  in  the  record,  or  in  reference  to  the  personnel  data  if  correction  is 
feasible.  Change  illegal  values  to  a  missing  data  code  as  a  last  resort.  If  a  sequence  of  data  for 
an  individual  appears  to  be  invalid,  the  respondent  may  have  placed  answer  codes  with  the 
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wrong  item  numbers.  If  the  sequence  is  not  correctable,  it  may  be  necessary  to  change  the  data 
from  that  page  of  the  survey  to  the  "missing  data"  codes. 

4.  Data  Analysis.  The  following  steps  in  data  analysis,  discussed  sequentially,  are  appropriate 
for  each  form  as  its  data  become  available  after  data  entry  and  cleaning.  After  Step  1 ,  not  all  the 
subsequent  steps  may  be  necessary,  but  each  step  provides  much  more  information  yield  from 
the  same  data.  With  this  type  of  exploratory  survey  data  analysis,  the  range  of  specific  analyses 
to  be  performed  is  very  large  (sec  below),  so  the  expected  outcomes  cannot  be  predicted  in 
advance. 

Step  1  -  Frequencies.  Examine  the  numbers  and  percents  for  the  full  set  of  responses  for  each 
item.  Does  the  distribudon  of  responses  make  sense?  If  not,  can  the  data  be  corrected  by  reference 
to  another  data  item,  so  that  aggregate  summary  data  make  sense  and  will  not  be  misleading  when 
reported?  The  frequencies  and  percents  for  many  items  are  likely  to  be  of  interest  in  themselves, 
such  as  the  types  of  training  experienced,  delivered  or  managed;  the  perceptions  of  problems 
encountered;  the  types  of  training  technology  used;  and  opinions  about  the  current  unit-level 
training  situation.  Graphs  for  presenting  such  ttems  would  be  desirable. 

Step  2  -  Create  Summary  Variables.  When  constructing  the  surveys,  items  were  created  to 
cover  each  step  of  the  ISD  process,  as  well  as  to  explore  potential  problem  areas  concerning  each 
task  in  delivering  and  managing  unit-level  training.  In  addition,  exploratory  visits  to  Andrews  and 
Randolph  AFBs  for  pilot  interviews  with  trainers,  training  managers,  and  trainees  helped  to 
delineate  the  processes  and  examples  of  problems  occurring.  This  conceptual  background  was 
used  to  ensure  coverage  of  the  training  process  and  will  form  the  basis  for  summary  variables  for 
analysis.  Attachment  1  provides  a  listing  of  all  survey  questions  as  they  apply  to  specific 
categories  of  the  ISD  process.  Those  items  that  are  demographic  in  nature  are  excluded. 

The  analysis  is  likely  to  be  clearer  and  simpler  to  understand  if  summarizing  and  categorizing 
variables  are  derived  at  this  point  and  added  to  the  data  file  being  analyzed.  These  variables  tend  to 
be  of  two  types  which  are  detailed  below:  1)  refined  sets  of  categories  to  be  used  in  later  cross- 
tabulations,  such  as  the  AFSC  job  category;  or  2)  continuous  variables  measuring  something  on  a 
numeric  scale,  such  as  a  summary  index  of  problems  perceived  with  a  specific  type  of  training 
Some  examples  of  such  variables  follow: 
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1 .  AFSC  categories  -  Create  categorical  variable  to  summarize  career  codes  with  a  limited 
number  of  categories.  The  actual  distribution  of  the  original  responses  should  be 
examined  before  deciding  on  the  categories,  so  that  a  minimum  of  about  50  respondents 
are  in  each  final  category.  Several  related  career  codes  may  need  to  be  grouped  together, 
in  order  to  yield  a  small  enough  number  of  categories  for  analysis. 

2.  Extent  of  automation  used  -  For  example,  for  the  training  managers,  create  a  summary 
categorical  variable  to  summarize  the  extent  of  automation  used  across  several  functions, 
such  as  scheduling,  documentation,  reporting  and  evaluation.  The  actual  distributions  of 
the  data  items  and  their  correlations  must  be  examined  to  determine  what  combinations 
may  be  useful  to  summarize  across  several  individual  items. 

3.  For  sets  of  numeric  items,  such  as  those  concerning  problems  in  scheduling  or 
documentation  among  training  managers,  scales  can  be  created  to  summarize  groups  of 
items  which  show  common  tendencies  among  respondents.  Use  of  such  scales  can 
simplify  the  reporting  of  results  for  these  lengthy  sets  of  items.  For  example,  a  single 
scale  might  summarize  the  responses  to  eight  items  for  training  managers  concerning 
problems  with  scheduling  of  training.  Use  factor  analysis  (principal  components)  or 
examine  the  matrix  of  correlation  coefficients  to  determine  which  items  "go  together" 
into  a  pattern  that  makes  sense  and  can  be  given  a  name  for  easy  interpretation.  Then 
combine  the  responses  for  that  set  of  items  (usually,  by  obtaining  the  mean  of  non¬ 
missing  values)  for  each  respondent  and  add  this  mean  value  for  each  respondent  as  a 
new  quantitative  variable  to  the  data  set. 

4  For  a  set  of  items  on  which  the  respondent  checks  off  "all  that  apply”,  a  useful  summary 
variable  would  be  a  count  of  the  number  of  items  that  each  respondent  checked  on  a 
specific  set.  For  example,  a  count  of  the  number  of  items  checked  on  an  item  concer¬ 
ning  the  types  of  tasks  performed  by  a  trainer  or  training  manager  would  summarize  the 
breadth  of  that  person's  role  in  training  activities. 

Step  3  -  Cross-Tabulations  and  Comparisons  of  Means.  These  analyses  help  to 
understand  whether  respondents  from  different  backgrounds  or  with  different  experiences  in  the 
Air  Force  tend  to  react  differently  to  the  survey  items.  A  cross-tabulation  is  a  table  for  examining 
the  relationship  of  one  categorical  variable  to  another  categorical  variable,  usually  using  the  X- 
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statistic  for  testing  the  presence  of  an  overall  relationship.  For  example,  do  trainees  from  different 
AFSCs  tend  to  experience  different  methods  for  job-related  training?  Do  training  managers  with 
versus  without  the  75xxx  code  as  their  primary  AFSC  tend  to  have  different  functions  in  managing 
training?  Do  they  use  technology  differently0 

A  comparison  of  means  is  used  to  examine  differences  among  subgroups  on  a  quantitative  item, 
such  as  items  scored  from  "strongly  agree"  to  "strongly  disagree,"  or  a  scale  combining  several 
such  items.  For  example,  do  training  managers  coming  from  different  types  of  squadrons  tend  to 
respond  differently  to  the  items  about  problems  with  documentation  or  scheduling?  Do  trainees 
from  different  AFSCs  have  different  opinions  about  the  training  technology  they  have  experienced0 

A  large  number  of  exploratory  analyses  of  this  type  are  feasible,  especially  using  the  summary 
variables  discussed  above.  The  results  of  the  most  interesting  or  surprising  of  these  analyses 
should  then  be  presented  in  graphs. 

Some  of  the  sub-groups  that  would  be  useful  to  examine  within  these  data  concerning  Squadron 
Level  Training  are  the  following: 

-  Types  of  units: 

-  AFSC  job  classes,  collapsed  as  needed  into  major  groupings; 

-  Training  managers  with  versus  without  the  75xxx  AFSC; 

-  Grade  level,  E-3  to  E-9  (as  relevant  for  each  survey  form); 

-  Major  Command  (MAJCOM); 

-  Years  of  active  duty,  categorized  as  less  than  1,1  to  2.  2  to  3,  3  to  4.  or  4  or  more  for 
trainees;  categonzed  as  !  to  4.  5  to  8.  9  to  12,  13  to  16,  or  more  than  16  for  trainers  and 
training  managers; 

-  Extent  of  civilian  education,  among  trainees. 
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-  Racial-ethnic  background,  among  trainees; 

-  Gender,  among  trainees; 

-  Role  in  the  training  process,  for  trainers  and  managers; 

-  Training  Managers  who  spend  most/ail  of  their  duty  time  on  training  activities  versus  those 
who  spend  little  of  their  duty  time  on  training; 

-  Trainees  who  have  experienced  different  types  of  training  technologies. 

Step  4  -  Correlational  Analysis  and  Multiple  Regression.  These  techniques  are  used  to 
understand  relationships  among  continuous  items,  and  to  analyze  more  complex  patterns  of 
relationships.  For  example,  one  might  want  to  examine  what  variables  are  related  to  a  tendency  to 
perceive  fewer  problems  with  a  particular  aspect  of  training,  such  as  the  use  of  specific  training 
methods,  belonging  to  certain  types  of  units,  the  amount  of  duty  time  one  spends  on  training,  and 
so  forth.  Since  these  surveys  are  intended  primarily  to  obtain  descriptive  information  about  current 
squadron-level  training  practices  and  perceived  needs  for  the  future,  the  items  have  not  been 
designed  to  test  particular  hypotheses  or  causal  models.  Nevertheless,  some  exploratory  analyses 
of  potential  relationships  are  likely  to  be  feasible. 

Specific  Analytic  Questions.  The  analytic  steps  outlined  above  all  involve  using  the  data 
from  these  three  groups  of  respondents  to  understand  the  current  extent  of  use  of  technologies  for 
unit-level  training,  and  respondents'  assessments  of  the  effectiveness,  reliability,  problems,  or 
other  applicable  dimensions  associated  with  that  aspect  of  training.  Interpreting  the  nature  of  the 
respondents'  perceptions  in  light  of  the  extent  of  their  experience  with  a  technology  will  provide 
information  for  making  inferences  concerning  a  match  between  future  needs  and  technologies. 
Most  sections  of  the  surveys  examine  a  specific  Topic  concerning  training,  such  as  OJT,  Ancillary 
Training,  or  management  functions  such  as  Scheduling,  Record-Keeping  or  Evaluation.  Most 
sections  use  the  following  general  logic  to  examine  that  Topic  area,  with  the  corresponding  specific 
analync  quesnons  (AQ): 

A.  What  training  has  been  experienced  or  which  sub-tasks  does  the  respondent  do.  in  regard  to 
this  Topic9 
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AQ  What  is  the  respondent's  role/expcnence  with  this  Topic? 

B .  What  is  the  respondent’s  scope  of  use  (e.g..  *7c  time  on  that  activity,  #  yea^  of  experience) 
with  that  Topic? 

C.  What  technologies/methods  have  been  used  for  this  Topic? 

AQ:  How  extensive  is  the  current  use  of  training  technologies  in  comparison  with  the  use  of 
other  methods  (manual,  one-on-one  instruction,  etc.)? 

AQ  Does  the  extent  of  use  of  technologies  versus  other  methods  differ  for  the  various 
Topics?  Differ  among  types  of  respondents? 

AQ  Does  the  extent  of  use  of  training  technologies  in  comparison  with  other  methods  differ 
by  the  respondents'  roles  or  scope  of  use  (i.e.,  items  A  &  B)? 

D.  What  are  the  respondents'  assessments  of  the  effectiveness,  reliability,  types  of 
problems,  etc.  concerning  that  Topic  and/or  their  use  of  technologies  for  it'1 

AQ  What  are  the  major  problems  (needs)  perceived  at  the  Squadron  level  with  local  level 
training?  Logically,  are  these  problems  that  are  likely  to  be  addressable  by  new  or 
changed  technologies? 

AQ:  Are  the  assessments  or  problems  (needs)  related  to  the  extent  of  use  of  a  specific 
method?  For  example,  do  those  who  have  experienced  mote  extensive  use  of  computerized 
training  technologies  tend  to  perceive  more  problems  with  them?  If  so,  this  may  suggest 
needs  for  work  on  technology  development,  to  make  the  technologies  more  appropriate  or 
more  reliable,  etc.  If  the  converse  occurs,  i.e.,  if  those  who  have  ltSi  experience  with 
computerized  technologies  tend  to  perceive  greater  problems,  this  ma  uggest  a  need  to  work 
on  implementation  problems  at  the  local  level.  Thus,  the  exact  nature  of  the  analyses  cannot 
be  specified  in  advance.  Final  analyses  will  depend  on  the  findings  from,  and  hypotheses 
suggested  by,  the  initial  analyses. 
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AQ:  Are  the  reported  assessments  or  problems  (needs)  related  to  the  respondent  s  role  or 
scope  of  use  on  that  Topic?  If  so,  these  findings  are  likely  to  suggest  hypotheses 
concerning  the  sources  of  squadron- level  training  problems  (needs). 

E.  Each  of  the  above  analyses  of  Topic  areas  may  also  differ  among  one  or  more  of  the  various 
sub-groups  listed  in  Step  3  above.  A  large  number  of  exploratory  data  tables  will  be 
examined  to  determine  whether  the  findings  for  a  Topic  differ  by  sub-group,  or  can  be 
safely  generalized  across  the  respondents  to  each  survey.  The  presentation  of  reported 
results  will  include  only  those  sub-group  analyses  that  show  meaningful  differences  among 
sub-groups. 

F.  A  further  section  of  the  trainers'  and  training  managers'  surveys  provides  a  scenario  of 
expanded  responsibilities  for  training  at  the  Squadron  level  in  the  future,  and  asks  a  number 
of  items  concerning  the  respondents’  expectations  for  further  technology  needs  in  the  future. 
To  the  extent  that  respondents  are  able  to  respond  thoughtfully  to  these  items,  they  will 
provide  rich  data  concerning  expected  future  needs  for  training  and  training  technologies  in 
case  residential  training  is  reduced. 

The  analysis  of  these  surveys  will  provide  a  detailed  view  of  the  current  status  of  training  at  the 
squadron  level  including  problems  in  the  training  process,  the  use  of  training  technologies  at  the 
squadron  level,  as  well  as  assessments  of  positive  and  negative  aspects  of  that  use.  The  use  of 
these  analyses  for  matching  future  "needs"  and  "technologies"  will  be  from  logical  interpretations 
of  the  survey  findings  in  relation  to  the  potential  uses  and  characteristics  of  technologies  -  both 
those  currently  available  and  feasible  future  developments.  For  this  logical  analysis,  additional 
information  about  the  nature  of  the  technologies,  gathered  through  archival  research,  will  be  used 
to  identify  appropriate  technologies  that  address  current  and  future  needs  for  squadron  level 
training. 
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Appendix  F 

Relating  Training  System  Needs  to 
Training  Technologies 
(Phase  I  Extract) 
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RELATING  TRAINING  SYSTEM  NEEDS 
TO  TRAINING  TECHNOLOGIES 

This  chapter  describes  the  process  by  which  the  needs  of  the 
unit-level  training  system  were  systematically  identified, 
documented  and  matched  to  HSD  training  technologies.  In  the 
previous  chapter,  researchers  identified  a  number  of  general 
problems  in  the  unit-level  training  system  for  PACAF  F-16 
maintenance  units.  Researchers  described  these  problems  with 
broad  narrative  statements  based  on  the  results  of  their  content 
analyses.  However,  these  broadly-defined  problems,  while 
insightful  in  and  of  themselves,  required  further  analysis  before 
they  could  be  related  to  training  technologies.  This  chapter 
documents  the  efforts  of  researchers  to  identify  the  specific 
training  system  functions  implicated  by  these  problem 
descriptions  as  being  deficient,  and  subsequently,  to  relate  them 
to  the  functions  of  HSD  training  technologies. 

The  impetus  for  this  line  of  Air  Force  research 
lies  in  the  certainty  that  force  structure  and  budgetary  levels 
will  continue  the  trend  of  systematic  reductions  and  short-term 
adjustments  to  immediate  fiscal  shortfalls.  This  activity 
constitutes  an  intermediate  step  in  the  development  of  a  long- 
range  S&T  Investment  Strategy  with  the  establishment  of  a 
framework  with  which  to  focus  the  emerging  technologies  against 
concrete  user  needs.  Matching  of  training  needs  and  technologies 
is  only  a  preliminary  step, .however,  since  this  phase  of  the 
research  initiative  examined  only  a  narrow  segment  of  the  Air 
Force  training  domain  (i.e.,  F-16  aircraft  maintenance  units 
deployed  in  the  Pacific  theater).  Accordingly,  suggestions  will 
be  made  for  further  research  that  will  attempt  to  generalize  the 
results  of  this  data  collection  across  a  larger  segment  of  the 
Air  Force's  vast  training  system. 

A.  Analytic  Framework:  ISD  Model 

A  review  of  available  training  literature  and  AF  training 
research  revealed  that  no  clear-cut  or  systematic  methodologies 
existed  for  identifying,  documenting  and  relating  training  system 
needs  to  training  technologies  (Carson,  Chambers  &  Gosc,  1984; 
Chenzoff  et  al . ,  1984  ;  Stephenson  &  Burkett,  1975).  Most  of  the 
research  in  this  area  has  focused  on  training  needs  assessment  and 
has  not  attempted  to  identify  and  relate  the  needs  of  the  training 
system  to  potential  technology  solutions. 

This  study  adopted  a  "training  system  needs  assessment" 
approach  as  opposed  to  a  traditional  training  needs  assessment 
approach.  The  distinction  between  the  two  is  important  here.  On 
the  one  hand,  a  training  needs  assessment  "provides  information  on 
where  training  is  needed,  what  the  content  of  the  training  should 
be,  and  who  within  the  organization  needs  training  in  certain  kinds 
of  skills  and  knowledge  (Ostroff  £>  Ford,  1989)."  This  type  of 
assessment  stresses  the  importance  of  three  interrelated 
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components:  organization,  task,  and  person.  A  training  system 
needs  assessment,  on  the  other  hand,  provides  information  on  which 
processes  or  functions  in  the  training  system  need  improvement. 
These  processes  can  be  broadly  stated  as  planning,  programming, 
management,  evaluation,  development  and  delivery  of  training. 
Consequently,  training  needs  assessment  would  be  one  process  within 
this  system. 

The  basic  analytic  framework  chosen  for  this  research  was  the 
Instructional  System  Development  (ISD)  model.  This  model  had 
several  features  which  made  it  appropriate  for  this  purpose. 
First,  the  ISD  model  covered  a  wide  range  of  training  activities, 
including  planning,  programming,  development,  delivery, 
management,  and  evaluation.  Moreover,  the  model  can  be  defined 
at  such  a  level  as  to  differentiate  between  the  unique  functions 
of  various  training  technologies.  Many  technologies,  while 
somewhat  similar,  are  intended  to  enhance  different  aspects  of 
the  same  general  training  area.  For  example,  two  technologies 
might  enhance  training  management  but  do  so  in  qualitatively 
different  areas,  such  as  resource  scheduling  and  student 
tracking.  A  broadly  defined  framework  would  not  capture  these 
nuances.  Fortunately,  the  ISD  model  allowed  us  to  describe 
training  technologies  and  research  at  a  meaningful  and 
analytically  useful  level. 

The  ISD  model  is  also  widely  accepted  within  the  AF  and 
other  training  communities,  although  the  exact  form  of  this  model 
is  a  matter  of  continuing  debate.  AFM  50-2  and  AFP  50-58,  in 
particular,  formed  the  basis  of  the  framework.  The  final 
version,  as  seen  in  figure  3,  was  the  result  of  repeated  reviews 
and  modifications  by  AL/HRT  ISD  experts.  Each  of  the  processes 
is  defined  in  Appendix  F. 

3.  Applying  the  Analytic  Framework 

The  process  of  matching  training  technologies  to  training 
system  needs  occured  in  three  steps.  In  the  first  step, 
researchers  developed  profiles  for  a  selected  group  of  training 
technologies  based  on  the  analytic  framework  shown  in  figure  3. 
The  "technology  profiles"  contained  the  specific  ISD  processes 
(i.e.,  training  functions)  which  were  enhanced  by  a  technology. 
Next,  researchers  examined  the  general  training  system  problems 
identified  during  analysis  of  the  interview  data  to  determine  the 
specific  ISD  processes  which  needed  improvement.  As  such,  the 
'training  system  problem  profiles"  contained  the  specific  ISD 
processes  which  needed  improvement  according  to  the  researcher's 
interpretation  of  the  general  problem  description.  In  the  final 
step,  the  technology  profiles  were  compared  to  the  training  system 
problem  profiles,  and  areas  of  overlap  and  non-overlap  were  noted. 
Following  is  a  discussion  of  the  procedures  and  results  of  this 
matching  process. 
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Figure  3.  Analytic  Framework  Based  On  Instructional  System 
Development  Model 
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Note.  Adapted  from  AF  Manual  50-2  and  AF  Pamphlet  50-58 
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(Figure  3  continued) 
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Training  Technology  Frofiles 

HSD  has  a  number  of  training  technologies  currently  under 
development  {Appendix  E).  The  impetus  for  training  technology 
research  comes  from  a  number  of  different  sources  including 
formal  requests  from  AF  organizations  as  well  as  self -initiated 
research  in  response  to  anticipated  needs.  Yet,  whatever  their 
source  is,  these  technologies  have  one  goal  in  common,  to  improve 
the  effectiveness  or  efficiency  of  one  or  more  parts  of  the 
training  system.  The  objective  of  this  step  of  the  matching 
process  was  to  develop  a  profile  for  each  of  the  selected  training 
technologies  by  identifying  those  parts  of  the  training  system 
which  they  were  intended  to  enhance.  In  other  words,  each 
technology  is  intended  to  add  to  the  knowledge  or  technology  base 
of  one  or  more  of  the  ISD  processes. 

The  term  "training  technology,"  as  it  is  used  here,  requires 
some  explanation.  Rousseau  defined  technology  as  "a  process  for 
transforming  physical  and  information  inputs  into  outputs  (1983, 
p.  225).”  As  such,  a  training  technology  can  refer  to  a  wide 
range  of  items,  including  hardware,  software,  research 
procedures,  data  collection  instruments,  and  analysis  tools.  In 
this  study,  researchers  were  primarily  concerned  with  research 
projects  and  not  necessarily  individual  training  technologies, 
per  se.  These  projects  may,  in  fact,  develop  and  integrate 
several  types  of  training  technologies  into  one  system  in  order 
to  achieve  a  specific  goal  or  meet  a  specific  need.  For 
simplicity  sake,  however,  a  project  will  be  referred  to  as  a 
training  technology,  whether  it  encompasses  a  single  technology 
or  a  system  of  integrated  technologies. 

The  most  reliable  sources  of  information  about  the  training 
technologies  were  project  managers  and  laboratory  technical 
directors.  These  people  possessed  the  most  accurate  and  up-to- 
date  knowledge  about  the  specific  projects  and  could  identify  the 
specific  training  functions  that  the  technologies  were  designed 
to  enhance.  Researchers  provided  the  laboratory  technical 
directors  who  were  responsible  for  technical  training-related 
research  with  the  ISD-technology  matrix  seen  in  Figure  4  and  asked 
them  to  indicate  '^elow  each  of  their  technologies  the  specific 
training  processes  oeing  enhanced  (i.e.,  the  knowledge  or  technical 
base  of  that  ISD  process  is  increased).  They  also  received  a  list 
of  definitions  clarifying  the  function  of  each  ISD  process.  The 
final  set  of  profiles  (Figure  4)  represents  the  collaborative 
efforts  of  the  technical  directors  and  the  project  managers  who 
oversaw  the  research  and  development  of  the  individual 
technologies . 

The  technology  profiles  presented  here  portray  the  intended 
final  form  of  the  technologies.  Since  these  technologies  are 
currently  under  development,  their  profiles  may  or  may  not  change 
with  time.  For  example,  the  Advanced  On-The-Job  Training  System 
(AOTS)  evolved  into  a  production  ready  technology,  the  Base 
Training  System,  which  provides  only  a  portion  of  the  training 
functions  of  the  original  technology  prototype.  It  went  from 
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Figure  4.  Profiles  Cataloguing  the  ISD 
Processes  Enhanced  By  Each 
Training  Technology 
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being  an  integrated  management,  evaluation,  development  and 
delivery  system  for  OJT  to  a  management-oriented  system. 
Nevertheless,  the  profiles,  as  presented  here,  provide  a  snap-shot 
of  the  current  direction  of  AF  training  technologies. 

Training  System  Problem  Profiles 

Technology  profiles  represented  one  side  of  the  equation 
in  the  matching  process.  On  the  other  side  were  the  profiles  for 
the  training  system  problems.  Their  purpose  was  to  systematically 
document  those  functions  of  the  training  system  which  researchers 
suspected  to  be  deficient  based  on  their  analysis  of  unit-level 
training  requirements  and  capabilities  (see  Table  6  a  summary  of 
the  training  problems).  The  same  ISD  analytic  framework  used  by 
technical  directors  and  project  managers  to  describe  their 
technologies  were  also  used  by  researchers  to  translate 
descriptions  of  training  system  problems  into  sets  of  specific 
training  system  needs.  The  results  of  the  analyses  are  shown  in 
Figure  5.  Solid  circles  represent  processes  where  evidence 
strongly  suggests  an  ISD  process  is  being  inadequately  performed  at 
the  unit  level  and  thus  needs  improvement.  Empty  circles  represent 
ISD  processes  where  evidence  inconclusively  suggests  improvements 
are  needed. 

Following  are  the  rationale  behind  construction  of  each  of 
the  problem  profiles: 

1.  Inappropriate  Training  Requirements.  This 
situation  strongly  indicated  a  deficiency  in  the  ISD  process 
entitled  "select  tasks  requiring  training."  Maintainers 
felt  that  some  of  the  globally  imposed  training 
requirements,  such  as  block  training,  were  entirely 
inappropriate  or  overtrained.  They  also  felt  that  there  was 
an  absence  or  lack  of  training  in  others  areas,  such  as 
CAMS,  troubleshooting,  basic  system  knowledge,  and  Chemical 
Warfare.  Moreover,  individuals  at  the  unit-level  suffered 
because  managers  and  supervisors  were  uncomfortable  or 
unable  to  make  decisions  on  training  content  for  any 
individual.  This,  in  turn,  negatively  impacted  the 
"management  of  training  requirements"  at  the  units.  The 
situation  described  here  strongly  suggests  that  the 
mechanisms  for  selecting  tasks  requiring  training,  at  the 
unit-level  and  higher,  were  either  flawed  or  unavailable. 

This  situation  may  also  reflect  deficiencies  in  the 
processes  for  "developing  task  listings"  and  "analyzing  job 
tasks."  For  example,  an  inaccurate  job  task  listing  can 
lead  to  inappropriate  training  if  it  includes  irrelevant 
tasks  or  to  a  lack  of  training  if  it  excludes  relevant 
tasks.  Moreover,  if  tasks  were  not  properly  analyzed,  they 
could  lead  to  over-  or  under-training.  Given  the  level  of 
data  and  primitive  state  of  the  framework,  however,  these 
conclusions  are  speculative  at  best. 
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(Figure  5  continued) 
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2.  Lack  of  Qualified  Instructors  and  OJT  Trainers.  This 
situation  was  largely  a  resource  problem,  in  terms  of  both 
quantity  and  quality.  The  lack  of  available  instructors 
(i.e.,  the  quantity  problem)  indicated  deficiencies  in  the 
ability  to  "select  appropriate  training  settings"  and 
"forecast  resource / logistic  requirements."  Cn  the  one  hand, 
there  was  a  failure  to  adequately  forecast  and  assign  the 
necessary  number  of  instructors  to  FTDs ,  MATs,  and  units 
(although  this  may  have  been  a  function  of  external 
constraints  such  as  manpower  shortages).  On  the  other  hand, 
this  situation  indicates  that  training  was  inappropriately 
allocated  to  training  settings  with  insufficient  resources. 
This  situation  may  also  indicate  an  inability  at  the  unit 
level  to  properly  "schedule  resources"  for  training. 

When  instructors  were  available,  many  of  them  appeared 
to  lack  technical  competency  and/or  teaching  abilities.  The 
latter  situation  strongly  indicates  that  a  problem  existed 
in  the  "selection  training  requirements"  for  instructors. 
This  conclusion  is  further  supported  by  the  apparent  lack  of 
available  courses  on  the  subject  of  teaching.  However,  the 
evidence  surrounding  the  former  problem,  i.e.,  lack  of 
technical  competency,  is  for  the  most  part  unclear  as  to  its 
implications.  It  may  indicate  potential  deficiencies  in  a 
number  of  training  processes,  but  further  investigation 
would  be  required  to  determine  which  ones  resulted  in  the 
lack  of  technical  competence  amongst  some  instructors. 

3.  Manpower  and  Personnel  Policies.  This  situation  does  not 
implicate  any  specific  training  process.  It  is,  in  fact,  an 
external  factor  (situational  constraint)  which  adversely 
affects  the  training  system  by  increasing  training 
requirements . 

4.  Enhanced  Hands-On  Training/OJT .  This  situation  is  quite 
broad  in  its  implications,  suggesting  improvement  to  all 
aspects  of  OJT  including  planning,  management,  evaluation, 
development  and  delivery.  The  emphasis,  however,  is  on 
"delivery"  and  the  need  for  "instructional  methods"  and 
"instructional  media"  which  accentuate  hands-on  skill 
training.  It  is  unclear  from  the  data  which,  if  not  all,  of 
the  subprocesses  in  the  selection  of  instructional  methods 
and  media  are  implicated  by  this  problem. 

5.  Lack  of  Training  Opportunities  Due  To  Improved  Weapon 
System  Technology.  This  situation  clearly  indicates  the 
need  for  instructional  developers  to  "select  instructional 
media"  which  will  enable  maintainers  to  train  on  tasks  which 
infrequently  occur  because  of  extremely  high  weapon  system 
reliability  rates.  This  also  indicates  a  need  for  an 
improved  "training  requirements  selection"  process  so  that 
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infrequently  performed  task  are  identified  for  recurring 
training . 

6.  Limited  Training  Time  Due  To  Operational  Commitments. 
This  situation  is  quite  complex  and  indicates  areas  needing 
improvement  within  training-  and  external  to  it.  External 
factors  which  limit  time  include  mission  requirements  and 
limited  manpower.  The  implication  for  training  is  to 
increase  the  efficiency  and  availability  of  training  through 
enhanced  "selection  of  instructional  methods”  and  "media” 
and  to  make  "delivery"  of  training  more 

accessible /convenient . 

7.  Lack  of  Enroute  Training.  The  circumstances  surrounding 
this  situation  point  to  problems  in  the  assignment  process 
and  deficiencies  in  the  "scheduling  of  students."  According 
to  interviewees,  the  procedures  for  scheduling  students  for 
enroute  training  were  slow  and  unreliable. 

8.  Inadequate  Feedback  Mechanisms  For  Evaluating  Training 
Courses.  This  situation  indicates  a  need  for  enhanced 
feedback  mechanisms  to  support  the  "maintenance  and  update 
of  the  instructional  system"  and  the  "evaluation  of  the 
external  validity  of  the  instructional  product." 

Maintainers  had  mixed  impressions  about  whether  or  not  their 
inputs  regarding  the  content  of  training  courses  were  being 
received  and,  when  received,  used  by  course  developers  to 
evaluate  and  update  their  courses. 

9.  Inadequate  Job  Performance  Measurement  Procedures.  This 
situation  points  to  deficiencies  in  the  "development  of  job 
performance  testing  procedures".  Maintainers  claimed  that  QA 
evaluations  were  unstandardized  and  inconsistent  across 
evaluators.  Moreover,  many  of  the  maintainers  and  some  QA 
personnel  felt  that  the  evaluations  were  not  a  good  measure  of 
a  unit's  capabilities  since  the  same  set  of  personnel  seemed 
to  be  evaluated  every  time  QA  visited  a  unit.  For  example, 
personnel  on  swing  shifts  rarely  were  evaluated.  QA 
evaluators  were  also  criticized  for  not  being  proficient 

in  the  tasks  they  evaluated.  This  problem,  however,  seemed 
to  be  a  temporary  one  brought  about  by  Rivet  Workforce 
training  requirements  and  should  dissipate  once  QA  personnel 
become  more  familiar  with  the  rivetized  tasks.  (QA  personnel 
also  performed  evaluations  of  student  performance  in 
conjunction  with  AMQP.  However,  it  is  unclear  from  the  data 
whether  or  not  these  training  evaluations  suffered  from  the 
same  lack  of  standardization  as  did  the  job  performance 
evaluations.) 

10.  Lack  of  TDT  Funding.  This  situation  is  predominantly  a 
reflection  of  recent  budget  cutbacks.  It  is  likely, 
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however,  that  training  managers  were  unable  to  forecast  the 
resource,  cost,  and  performance  impacts  of  their  decisions 
and,  therefore,  provided  relied  on  OJT  rather  than  sending 
someone  TDY  to  school.  This,  in  turn,  indicates  training 
settings  were  inappropriately  selected  (i.e.,  selection  by 
default).  For  example,  training  someone  on  a  certain  task 
or  set  of  tasks  in  OJT  may  be  more  costly  in  terms  of  labor 
hours  and  resources  than  sending  that  person  TDY. 

11.  Implementation  Problems  with  Hew  Training  Media.  This 
situation  pertains  to  IVD  technology,  in  particular,  and 
indicates  that  resources  were  not  properly  identified  and 
provided  to  support  IVD  use  and  implementation  at  the 
unit-level.  Moreover,  training  personnel  did  not  know  when, 
where,  or  how  to  use  that  particular  media. 

The  interview  data  and  findings  from  the  content  analysis 
strongly  support  the  existence  of  deficiencies  in  a  large  number  of 
training  processes.  These  deficiencies  represent  deficiencies 
with  the  unit-level  training  ystem  as  a  whole  and  are  not 
necessarily  confined  to  the  units  themselves.  For  example,  many  of 
the  processes  at  the  front-end  of  the  framework  (i.e.,  "Analyze 
System  Requirements";  "Define  Education  and  Training 
Requirements";  "Develop  Tests  and  Objectives";  "Plan,  Develop  and 
Validate  Instruction")  are  often  accomplished  by  organizations 
outside  the  unit-level.  The  results  of  these  activities, 
however,  are  nonetheless  intimately  linked  to  the  unit-level 
training  system.  It  is  important  to  understand  unit-level 
training  as  a  system  and  not  as  an  isolated  activity. 

Table  7  contains  a  summary  of  the  specific  ISD  processes 
implicated  by  the  problems  described  in  previous  chapters.  Only 
those  processes  which  are  strongly  supported  by  the  data  are 
listed  here.  The  degree  to  which  each  of  the  processes  is 
deficient  is  unclear  since  no  metric  existed  for  determining  the 
severity  of  each  problem.  Based  on  the  number  of  problem 
descriptions  (Table  6)  which  implicated  a  particular  process,  it 
would  appear  that  "Selection  of  Tasks  Requiring  Training", 
"Selection  of  Instructional  Media",  "Selection  of  Instructional 
Methods",  and  "Dt  very  of  Training"  are  the  processes  with  more 
pronounced  deficie.'  :ies.  However,  such  a  finding  would  be 
extremely  tentative. 

The  results  of  this  analysis  should  be  interpreted  cautiously 
for  several  reasons.  First,  it  is  likely  that  other  researchers 
would  have  differing  interpretations  of  the  problems  described  in 
this  paper,  and  therefore,  differing  profiles  and  conclusions. 
Second,  since  instructional  development  is  an  integrated  process, 
it  becomes  difficult  to  implicate  any  single  process  as  being 
deficient.  Problems  may  result  fiom  weaknesses  in  several 
processes,  or  the  deficiencies  in  one  process  may  overshadow  or 
mask  deficiencies  in  others.  Finally,  the  results  are  limited  by 
the  specificity  and  scope  of  the  Phase  I  SLT  data.  Researchers 
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developed  the  analytic  framework  used  in  this  analysis  aft^r  data 
collection  had  already  been  completed.  Consequently,  this 
framework  did  not  guide  the  data  collection  effort.  Data 
collection  was  instead  guided  by  a  similar,  yet  much  broader, 

Table  7 

Summary  of  Training  Processes  Implicated  bv  SLT  Phase  I  Findings 


1)  Selection  of  Tasks  Requiring  Training? 

2)  Selection  of  Appropriate  Training  Settings; 

3)  Forecast  of  Resource/Logistics  Requirements; 

4}  Development  of  Jod  Performance  Testing  Standards; 

5)  Selection  of  Instructional  Methods; 

6)  Selection  of  T astructional  Media; 

7)  Delivery  of  Instruction; 

P;  Scheduling  of  Students? 

9)  Scheouling  of  Resources; 

10)  Management  of  Individual  Training  Requirements? 

11)  Maintenance  and  Update  of  Instructional  Systems;  and, 

12)  Evaluation  of  External  Validity  of  Instructional  Product. 


framework  than  the  ISD  model  provided.  This  limited  the  ability  of 
researchers  to  identify  accurately  the  specific  training  system 
needs  implied  by  the  data.  They  were,  for  example,  unable  to 
identify  the  specific  ISD  processes  within  media  and  method 
selection  implicated  by  the  problem  descriptions.  This  inability 
was  largely  due  to  the  lack  of  specificity  in  the  data  collected. 
Su-h  a  situation  could  have  been  avoided  if  data  collection  had 
been  focused  along  the  lines  of  the  ISD-based  analytic  framework. 

Training  Needs-Technology  Match 

The  final  step  in  the  matching  process  involved  a  comparison 
of  the  technology  and  training  system  deficit  profiles.  The 
purpose  of  such  a  comparison  is  to  reveal  where  training  system 
needs  are  being  addressed  by  AF  training  research  and  where  they 
are  not.  The  results  of  the  matching  process  can  be  seen  in 
Figure  G.  The  shaded  areas  represent  processes  which  were 
implicated  as  being  deficient  from  earlier  analyses.  An  overlap 
{or  ’’hit")  exists  when  solid  circle  appears  in  the  shaded  area. 
This  indicates  that  the  deficient  ISD  process  is  addressed  b\  a 
training  technology.  The  mere  fact  that  an  overlap  exists, 
however,  is  not  confirmation  in  and  of  itself  that  a  system  need 
is  being  addressed.  Since  the  analytic  framework  used  in  this 
matching  process  is  very  basic  and  only  considers  function, 
researchers  have  to  consider  other  issues  such  as  the  context 
surrounding  a  training  need  (domain)  and  the  degree  to  which  a 
technology  addressed  this  context  in  addition  to  the  basic  overlap. 
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(Figure  6  continued) 
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VAUDtTC  COMPUTE  ST511M 


(Figure  6  continued) 
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The  results  of  this'  matching  process  are  discussed  in 
the  remainder  of  this  section,  focusing  on  each  of  the  processes 
listed  as  deficient  in  Table  7  and  their  relationship  to  AF 
training  technology  research. 

The  ISD  process  entitled  "select  tasks  requiring 
training"  was  implicated  as  being  deficient  several  times  in  the 
analysis  of  training  system  problems.  Figure  6  indicates  that 
four  technologies  (i.e.,  projects)  enhance  the  knowledge  or 
technology  base  for  the  selection  of  training  requirements, 
including  the  Advanced  OJT  System  (AOTS)/BTS,  the  Comprehensive 
Occupational  Data  Analysis  Package  {CODAP),  Job-Aiding/Training 
Allocation  Technologies  (JATAT),  and  Training  Evaluation.  Each 
of  these  technologies  is  intended  to  enhance  the  training 
requirements  selection  process  in  a  slightly  different  manner. 
Unfortunately,  the  analytic  framework,  as  it  is  currently 
defined,  does  not  distinguish  between  these  subtle  differences. 
Even  if  it  could,  the  level  of  specificity  of  the  data  would 
preclude  a  more  in  depth  comparison  of  needs  and  technologies. 
Suffice  it  to  say  that,  based  on  the  interview  data,  resources 
for  accurately  selecting  training  tasks  need  to  exist  at  all 
levels  of  the  training  system.  This  will  enable  training 
managers  to  tailor  training  requirements  to  their  particular 
situations . 

Deficiencies  in  the  processes  for  "selecting  appropriate 
training  settings"  and  "forecasting  resource  logistic 
requirements"  appear  to  be  closely  related.  The  reason  being 
that  forecasting  resource  and  logistic  requirements  was  an 
important  step  in  determining  appropriate  training  settings. 

In  other  words,  the  appropriateness  of  a  training  setting  was 
largely  dependent  upon  the  adequacy  of  its  resources.  Three  of 
the  training  technologies,  the  AOTS,  CBT  Selection  Advisor,  and  the 
Training  Decisions  Modeling  Technologies  (TMDT ) ,  provide  a 
capability  for  forecasting  resource  and  logistic  requirements. 
However,  only  one  of  these  technologies,  the  TDMT ,  goes  the 
additional  step  and  enhances  the  process  for  selecting 
appropriate  training  settings  based  on  the  forecasted  resource 
and  logistic  requirements. 

At  a  very  broad  level,  the  process  for  "selecting 
instructional  methods"  was  implicated  by  two  of  the  apparent 
training  system  problems.  Several  technologies,  including  the 
Advanced  Instructional  Design  Advisor  (AIDA),  Basic  Job  Skills 
(BJS),  CBT  Selection  Advisor,  and  Training  Evaluation,  address 
some,  if  not  all,  aspects  of  this  deficiency.  Unfortunately,  as 
previously  stated,  researchers  could  not  discern  from  the  data 
which  of  the  subprocesses  needed  improvement,  and 
therefore,  it  is  unclear  which  technologies  truly  address  the 
deficiency.  The  only  exception  to  this  would  be  AIDA  which 
addresses  each  aspect  of  method  selection. 

The  CBT  Selection  Advisor  is  the  only  technology  which 
addresses  the  deficiencies  in  "selecting  instructional  media". 

It  would  appear  that  this  technology  fully  addresses  the 
deficiencies  in  media  selection,  but  appearances  can  be 
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deceiving.  The  overlaps  shown  in  Figure  6  simply  indicate  that  a 
technology  enhances  the  same  type  of  functions  which  have  been 
identified  as  deficient.  A  closer  examination  of  the  situation 
reveals  that  the  technology  in  question  is  intended  to  enhance 
media  selection  and  use  within  one  specific  context  —  computer- 
based  training.  The  deficiencies  indicated  by  the  Phase  I 
research  appear  to  encompass  media  selection  at  a  more  general 
level;  that  is,  it  is  not  content  specific.  Training  developers 
and  manager  need  help  identifying  training  media,  CBT  or  other, 
suitable  to  the  unit-level  environment.  Whether  or  not  the  CBT 
Selection  Advisor  can  be  adapted  for  media  selection  in  general 
is  unknown. 

Deficiencies  in  the  "determination  of  resource  and  funding 
requirements'*  for  IVD  instruction  contributed  to  the  poor 
implementation  of  this  category  of  instruction  at  units.  The 
AOTS ,  CBT  Selection  Advisor,  and  TDMT  are  possible  solutions  to 
this  deficiency,  however.  These  technologies  would  enable 
training  developers  to  determine  the  exact  resource  and  funding 
requirements  necessary  to  support  their  instruction  prior  to 
actually  implementing  it. 

Training  delivery  is  the  most  salient  process  in  the 
instructional  system  and  is  the  culmination  of  all  the  other 
processes.  It  is,  if  you  will,  "where  the  rubber  meets  the 
road."  Of  the  fifteen  training  technologies  of  concern  in  this 
study,  almost  half  (i.e.,  seven}  of  them  are  intended  to  enhance 
the  delivery  of  instruction.  They  do  so  through  a  variety  of 
computer-based  and  intelligent  tutoring  systems.  However, 
deficiencies  which  appear  in  the  delivery  process  are  often  the 
result  of  deficiencies  in  other  processes.  While  the  interview 
data  suggests  that  the  delivery  of  instruction  needs  improvement 
to  overcome  problems  in  the  unit-level  environment  such  as 
limited  training  time,  it  is  likely  that  the  delivery  pioblem 
would  be  solved  if  other  processes  (e.g.,  selection  of 
instructional  media)  were  improved. 

All  but  one  of  the  processes  within  the  instructional  support 
area  were  identified  as  deficient.  These  processes,  along  with 
instructional  delivery  and  evaluation,  are  perhaps  the  most 
germane  tc  oper<.'  ional  units  (as  opposed  to  the  training  planning, 
programming  and  c-evelopment  functions  which  typically  occur 
outside  the  units  themselves).  According  to  the  data  collected 
at  PACAF  F-16  maintenance  units,  the  scheduling  of  both  resources 
(i.e.,  instructors)  and  students  (i.e.,  for  enroute 
training)  was  deficient.  The  AOTS/BTS  and  Instructional  Support 
System  (ISS)  are  intended  to  address  this  type  of  deficiency.  In 
fact,  the  AOTS/BTS  incorporates  ISS  technology  into  itself. 

Deficiencies  in  the  "management  of  individual  training 
requirements"  also  degraded  the  instructional  support  capabilities 
of  unit-level  training.  This  situation  is  very  much  related  to  the 
problem  surrounding  "selection  of  tasks  requiring  training"  and 
may,  in  fact,  only  be  a  consequence  of  this  deficiency.  In  this 
case,  however,  training  managers  were  unable  or  unwilling  to 
tailor  training  requirements  to  the  needs  of  individuals  at  a 
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job-site-  as  opposed  to  the  needs  of  an  AF  specialty  or  one  of  its 
shreds.  The  AOTS,  once  again  purports  to  enhance  the  process  for 
managing  individual  training  requirements.  It  is  also  likely  that 
the  capabilities  found  in  CODAP,  JATAT,  and  Training  Evaluation 
are  applicable  to  this  process  {just  as  they  are  to  "selection  of 
tasks  requiring  training"). 

Instructional  support  also  suffered  from  deficiencies  in  the 
"maintenance  and  update  of  the  instructional  system."  More 
specifically,  interviewees  felt  that  training  courses  were  not 
being  updated  or  modified  based  on  feedback  from  the  field  (via 
follow-up  surveys,  course  critiques,  and  informal 
communications).  Several  technologies  offer  potential  solutions 
to  this  problem  by  strengthening  the  feedback  loop  between  the 
field  and  the  course  developers,  instructors  and  administrators. 
These  technologies  include  AOTS,  CODAP,  and  the  Integrated 
Maintenance  Information  System  (IMIS).  Whether  or  not  the 
information  provided  by  these  technologies  is  actually  used  to 
update  courses  is  a  separate  question  beyond  the  scope  of  this 
research. 

Finally,  the  deficiencies  in  the  feedback  mechanisms  just 
described  also  indicate  deficiencies  in  the  process  for 
"evaluating  the  external  validity  of  instruction."  The  fact  that 
some  interviewees  felt  courses  were  not  being  updated  to 
reflect  the  realities  of  the  operational  environment  also  indicates 
that  the  external  validity  of  some  courses  is  questionable  (at 
least  in  terms  of  their  face  validity).  The  AOTS  and  JPM 
technologies  are  intended  to  improve  this  process  in  the  training 
system.  The  information  generated  by  these  technologies  should 
allow  course  developers  to  evaluate  the  degree  to  which  their 
courses  teach  the  skills  and  knowledges  relevant  to  the  work 
environment  including  appropriateness  of  course  content  and 
transfer  of  training. 

D.  Discussion  of  Matching  Process 

At  first  glance,  there  would  appear  to  be  no  apparent  gaps 
in  the  AF  training  technology  research.  Each  of  the  ISD  processes 
that  researchers  found  to  be  deficient  in  the  unit-level  training 
system  for  PACAF  F-16  maintenance  units  are  in  some  way  addressed 
by  a  training  technology.  The  only  area  which  may  be  overlooked 
is  that  of  training  media  selection.  Currently,  only  one 
technology,  the  CBT  Selection  Advisor,  addresses  this  problem, 
but  it  does  so  from  a  CBT  perspective  only. 

The  training  system  needs  and  technology  match  also  reveals 
areas  where  technologies  are  focused  but  which  were  not  implicated 
by  the  data  as  being  deficient.  However,  this  does  not  mean  that 
the  technologies  are  "misguided."  First  of  all,  this  study  was 
limited  to  PACAF  F-16  maintenance  units.  The  training 
technologies,  on  the  other  hand,  are  being  developed  for  a  broader 
AF  application.  Second,  while  only  certain  processes  were 
implicated  as  deficient  at  the  unit  level,  improvements  made  to  any 
of  the  processes  are  potentially  beneficial  to  unit-level  training. 
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Finally,  ‘the  data  collection  instruments  used  in  this  study  were 
not  based  on  the  analytic  framework  as  seen  in  figures  4,  5,  and  6. 
Consequently,  there  is  some  degree  of  error  in  the  results 
presented  here. 

Moreover,  the  framework  presented  in  this  report  has  a 
number  of  limitations  which  should  also  be  acknowledged.  It  was 
developed  specifically  for  describing  and  relating  training 
technologies  to  training  system  needs  at  a  very  basic  level;  that 
is,  it  merely  shows  where  the  functions  of  training  technologies 
match  deficiencies  in  the  training  system.  The  framework  does 
not,  for  instance,  specify  the  theoretical  principles, 
procedures,  and  rules  for  developing  and  operationalizing 
constructs,  and  for  analyzing  and  interpreting  data  (see  Ostroff 
&  Ford,  1989).  Nor  does  the  framework  explicitly  address  the 
issue  of  training  domains;  i.e.,  the  specific  contextual  areas 
with  which  training  technologies  or  deficits  are  associated,  such 
as  aircrew,  maintenance,  and  the  like. 

Nevertheless,  the  analytic  framework  represents  an 
important,  although  preliminary  step  toward  development  of  a 
systematic  process  for  identifying  and  describing  training  system 
needs,  cataloguing  training  technology  functions,  and  relating  the 
two.  Future  work  in  this  area  would  greatly  benefit  from  the 
theoretical  developments  found  in  training  needs  assessment 
research.  Ostroff  and  Ford  (1989),  for  example,  presented  a 
model  of  needs  assessment  based  on  a  levels  perspective. 
According  to  this  model,  training  needs  occur  at  three  levels: 
organizational,  subunit,  and  individual.  Each  of  these  levels  of 
analysis  has  implications  for  construct  development, 
operationalization,  and  interpretation.  It  is  likely  that  these 
concepts  would  also  be  applicable  to  training  systems  needs 
assessment  should  research  continue  in  this  area. 

In  summary,  the  data  indicates  that  training  system  needs 
are  being  addressed  by  training  technologies.  These 
relationships,  however,  deserve  a  more  in  depth  examination. 
Researchers  must  go  beyond  the  question  of  whether  or  not 
technologies  are  functionally  matched  to  deficiencies.  They  must 
examine  issues  such  as  the  degree  to  which  a  technology  meets  a 
need,  the  seriousness  of  a  training  system  need,  and  the  context 
surrounding  the  deficiencies.  Such  an  examination  would  reveal 
specific  areas  where  training  technologies  could  be  focused  in 
order  to  increase  their  applicability  to  unit-level  training. 
Before  this  can  take  place,  researchers  must  first  advance  the 
framework  which  has  guided  the  analysis  described  in  this 
chapter . 
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Appendix  G 

Survey  Respondents'  and 
Interviewees'  Demographics 


339 


Appendix  H 

Science  and  Technology 
Investment  Strategy 
Checklist  Guide 
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SCIENCE  AND  TECHNOLOGY  INVESTMENT  STRATEGY 
CHECKLIST  GUIDE 


This  Guide  provides  “how-to”  guidance  for  accomplishing  the  Science  and 
Technology  (S&T)  Investment  Strategy  Checklist  The  checklist  is  intended  to  structure 
the  research  process  to  ensure  that,  at  a  minimum,  certain  considerations  affecting 
selection  of  a  successful  S&T  investment  strategy  are  addressed. 

The  S&T  Investment  Strategy  Checklist  is  designed  to  be  used  by  all  members  of 
the  research  team  throughout  the  life  of  the  research  project.  It  should  be  updated 
continuously  as  the  research  effort  dictates,  and  critically  reviewed  at  periodic  team 
meetings  chaired  by  the  project  director.  Individuals  must  keep  in  mind  that  actions  they 
take  may  affect  actions  being  taken  by  others,  thus,  communication  between  team 
members  throughout  the  process  is  critically  important. 

The  checklist  is  not  intended  as  a  repository  for  all  research  data/information. 
Rather,  it  is  a  means  to  ensure  all  appropriate  actions  are  anticipated  and  accomplished, 
with  references  to  where  pertinent  data/information  can  be  found. 

Lessons  learned  in  the  pursuit  of  a  relatively  recent,  large-scale  research  and 
development  project  are  particularly  pertinent  to  efforts  of  this  nature.  In  the  conduct  of 
this  project,  it  was  determined  that,  to  be  successful,  one  should: 

•  address  recognized,  real,  and  substantial  needs 

•  with  realistic  objectives 
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•  using  feasible  enabling  technologies 

•  applied  in  iterative,  rapid  prototyping,  innovative  approaches 

•  that  make  frequent  use  of  concrete  demonstrations 

•  with  customer  participation  and  high-level  customer  support 

•  in  a  risk-tolerant  research  and  development  environment 

•  with  competent  people 

•  organized  into  a  development  team  with  appropriate  leadership 

It  would  be  beneficial  to  use  these  lessons  learned  as  guiding  principles  as  the 
attached  checklist  is  used  in  developing  an  investment  strategy. 
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The  checklist  is  organized  by  the  following  categories  and  sub-categories: 

1.  THE  PROBLEM 

1.01  Problem/Opportunity  Identification 
1.02  Characteristics  of  the  Product(s)  to  be  Produced 
1.03  Externally  Generated  Priorities 
1.04  Internally  Generated  Priorities 

2.  THE  TECHNOLOGIES 

2.0 1  Identification  of  T echnologies 

2.02  Opportunity  Costs 
2.03  Trade-off  Studies 
2.04  Risk  Assessment 
2.05  Software  Supportability 
2.06  Hardware  Supportability 
2.07  User  Training 
2.08  Flexibility 

3.  PROGRAM  PLANNING 
3.01  Coordination/Interface 
3.02  Funding 

3.03  Staffing 

3.04  Facilities 

3.05  Time-line 

3.06  Contracting  (if  used) 

4.  TEST  AND  EVALUATION 
4.01  System  Performance 
4.02  User  Acceptability 
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5. 


IMPLEMENTATION/DEPLOYMENT  PLANNING 
5.01  Cost-Effectiveness  Analysis  for  Deployment 
5.02  Implementation  Management 
5.02  Implementation  Resources 
6.  DOCUMENTATION  &  FOLLOW-UP 

6.0 1  Documentation  of  Development  Process 
6.02  Documentation  for  Technology 
6.03  Follow-up 
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The  Reference  Document  contains  a  detailed  discussion  of  each  of  the  categories 
and  subcategory  of  questions.  This  guide  describes  the  structure  of  the  checklist  and  the 
intended  use  of  each  column.  The  checklist  is  structured  as  follows: 

HEM!  HEM  ISSUES,  ACTION  QPE_  SUSPENSE  STATUS 

Although  an  attempt  has  been  made  to  array  the  categories  in  a  generally 
sequential  manner  for  most  processes,  there  will  be  times  when  it  makes  perfectly  logical 
sense  to  address  categories  or  issues  simultaneously,  or  seemingly  “out  of  sequence’*.  For 
example,  information  needed  to  respond  to  subcategory  3.02  (Funding)  might  be  readily 
available,  whereas  that  required  to  respond  to  subcategory  2.02  (Opportunity  Costs)  may 
very  well  take  more  time  to  develop  or  obtain. 

Following  each  category/subcategory  is  a  series  of  questions  to  be  addressed. 
These  preset  questions  are  designed  to  address  a  generic  S&T  investment  strategy.  Those 
which  do  not  apply  to  a  particular  project  should  be  annotated  with  “N/A”  in  the  “Issues” 
column.  Other  questions  can  be  added  to  tailor  the  analysis  to  the  specific  situation  or 
requirements. 

Note  that  all  questions  are  structured  to  illicit  a  “yes”  or  “no”  response. 
Dependent  upon  the  response,  issues  may  surface  which  must  be  addressed  before  the 
investment  strategy  is  pursued  further.  An  issue  is  a  potential  or  actual  deficiency 
identified  at  any  time  in  the  process  that,  to  a  greater  or  lesser  degree,  would  affect 
successful  pursuit  of  the  investment  strategy.  For  example,  if  the  answer  is  “yes”  to 
question  3.02.03  (Are  allocated  funds  adequate?),  no  issue  exists  and  a  notation  should 
be  made  under  the  “Issues”  column  which  would  provide  an  audit  trail  to  the  rationale  or 
documentation  which  supports  the  adequacy  determination.  However,  if  the  answer  is 
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“no,”  this  issue  may  be  a  “show-stopper”  if  not  resolved,  and  an  appropriate  statement  of 
the  issue  should  be  place  in  the  “Issues”  column.  Only  those  questions  which  are 
considered  not  appropriate  for  this  particular  review  should  be  accompanied  with  “N/A” 
in  the  “Issues"  column.  All  other  questions  should  be  acompanied  with  an  appropriate 
annotation  in  the  “Issues”  column,  whether  answered  “yes  or  “no”.  Although  the  use  of 
a  “do  not  know”  response  should  be  minimized,  it  is  recognized  that,  at  times,  this  may 
be  the  only  appropriate  response.  In  this  case,  it  is  particularly  important  that  a  clear 
statement  of  the  issues(s)  be  documented,  and  immediate  action  be  taken  to  either  resolve 
the  issue,  or  assign  action  to  someone  to  resolve  it 
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For  those  issues  identified,  actions  are  required  in  efforts  to  satisfactorily  resolve 
them.  These  actions  should  be  documented  in  the  “Action”column.  In  the  above 
example,  an  appropriate  action  may  be  to  “Obtain  adequate  funds,”  Request  adequate 
funds,”  or  “Reprogram  funds  from  a  lower  priority  project." 

For  those  issues  for  which  action  is  required,  a  person  or  office  should  be  assigned 
responsibility  for  accomplishing  the  action,  and  documented  in  the  “OPR”  column. 
Although  there  may  be  legitimate  reasons  for  reflecting  more  than  one  person  or  office  as 
being  responsible,  such  practice  should  be  avoided  as  much  as  possible.  Someone  with 
the  proper  authority  should  be  assigned  the  overall  responsibility  for  resolving  the  issue, 
and  be  held  accountable. 

In  order  to  keep  a  project  moving  and  on  schedule,  demanding  (but  realistic) 
“Suspense”  dates  for  resolution  of  issues  should  be  established  and  documented. 
Individual  suspenses  must  conform  to  the  project  timeline(s)  addressed  in  subcategory 
3.05  (Timeline).  If  a  suspense  is  missed,  or  anticipated  to  be  missed,  the  OPR  should 
document  the  reason  under  the  "Status"  column,  and  a  new  suspense  date  should  be 
assigned  by  the  Project  Director  and  indicated  in  the  “Suspense”  column. 

“Status”  codes  can  take  many  forms.  “Open”  and  “closed”  may  be  sufficient  for 
some  projects,  but  a  wider  array  may  be  more  appropriate  for  others.  Any  coding  scheme 
should  work  as  long  as  terms  are  well  defined.  If  not  clearly  defined,  terms  such  as 
“Open”  and  “Pending”  may  have  a  different  meaning  for  some  people,  but  the  same  for 
others. 
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1.0  THE  PROBLEM 

The  driving  force  behind  nost  research  in  science  and  technolog/  is  generally  the 
effort  to  solve  a  problem  or  take  advantage  of  a  research  oppor  .-unity.  Problem- 
driven  research  tends  to  respond  to  specific  needs,  such  as  reduced  training  time  or 
improved  overall  performance  in  a  certain  specialty.  Opportunity-driven  research, 
on  the  other  hand,  is  generally  guided  by  die  possibility  of  enhancing  an  activity 
beyond  its  current  capabilities,  or  the  hope  of  creating  an  entirely  new  activity. 
Initial  research  in  virtual  reality  is  a  prime  example  of  an  entirely  new  activity 
which  takes  advantage  of  breakthroughs  in  technology  to  enhance  training 
capabilities  far  beyond  the  potential  of  existing  simulators. 

Another  distinction  made  between  types  of  research  is  the  division  between 
theoretical  and  applied  research.  Both  theoretical  research  (research  directed 
towards  adding  to  die  general  body  of  knowledge  on  a  particular  topic)  and  applied 
research  (research  directed  towards  producing  immediately  useful  products  or 
information)  can  also  be  either  problem  or  opportunity  based.  For  example,  a 
problem  such  as  accurately  determining  how  children  learn  to  apply  the  rules  of 
language  would  be  classified  as  theoretical  research,  because  the  results  may  not 
have  any  immediate  practical  significance.  However,  in  the  future,  those  results 
may  well  lead  to  solutions  of  important  real  world  problems.  Applied  research  can 
start  from  the  perspective  of  addressing  a  problem  or  taking  advantage  of 
breakthroughs  in  one  area  to  further  existing  research  in  another. 


1.01  Problem/opportunlty  Identification 

1.01.01  Has  the  reason  for  this  Investigation  been  Identified  and 
documented? 

Without  a  clear  and  compelling  reason  for  conducting  research,  the  likelihood  of 
gaining  funding  and  organizational  support  is  significantly  reduced.  For  all 
research,  the  formulation  of  an  initial  statement  presenting  the  rationale  for  the 
investigation,  and  a  clear  research  question  or  hypothesis,  is  critical.  A  clear 
presentation  is  one  that  states,  in  the  simplest  terms  possible,  what  the  research  is, 
and  why  it  should  be  carried  out.  This  statement  should  include  the  current 
condition  or  problem  that  can  be  improved,  and  the  way  in  which  the  research  will 
improve  that  condition  (what  positive  effect  the  research  will  have  on  the  current 
condition).  This  improvement  could  be  an  increase  in  knowledge  or  a  change  in 
procedure.  When  appropriate,  a  statement  of  the  rationale  for  research  should 
include  an  explanation  of  what  could  happen  if  the  research  is  not  conducted.  This 
could  be,  perhaps,  that  the  current  undesirable  situation  will  continue,  become 
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LES^g  RESEARCH! 

1.  A  real,  identified  need  exists. 

2.  A  theoretical  problem  to  overcome. 

3.  The  opportunity  to  significantly  improve  an  approach  or 
outcome,  even  when  no  problem  or  need  is  currently  recognized. 

Furthermore,  the  researcher  must  make  sure  that  the  reason  for  the  research  is 
specific  to  the  needs  of  the  Air  Force.  When  preparing  a  statement  of  rationale  for 
research,  it  is  sometimes  helpful  to  use  documentation  from  sources  outside  of  your 
own  organization.  This  could  be  information  from  previous  studies,  or  statements 
from  agencies  or  commands  that  would  benefit  from  the  research. 


Documenting  the  goals  of  an  investigation  is  a  good  way  to  maintain  a  strong  focus, 
and  to  clear  up  any  ambiguity  or  confusion  that  may  occur  later.  This  document  can 
and  should  be  referred  to,  periodically,  as  a  tool  for  evaluating  the  direction  of  the 
project. 

The  following  section  provides  detailed  information  on  how  to  identify  the  reason 
for  your  investigation  by  creating  a  workable  "statement  of  problem"  in  the  form  of 
a  research  question  or  a  hypothesis.  A  dear  understanding  of  the  problem  that  is 
being  studied  is  the  foundation  of  every  solid  research  project— by  defining  the 
problem  to  everyone's  satisfaction,  the  investigative  efforts  will  be  sharply  focused 
and,  consequenfly,  more  productive. 


STATEMENT  PF  PROBLEM 

In  any  given  research  project,  the  first  step  should  always  be  the  statement  of  a 
problem,  and  the  establishment  of  either  a  research  question  or  a  hypothesis.  As 
mentioned  earlier,  the  former  is  more  common  in  applied  research  (research  that  is 
directed  towards  solving  a  problem)  and  the  latter  is  more  common  in  pure  research 
(research  directed  at  adding  to  the  general  body  of  knowledge  on  a  given  subject). 
Since  the  Air  Force  tends  to  be  more  concerned  with  applied  research  issues,  we  will 
primarily  focus  on  the  formulation  cf  a  research  question.  However,  the  following 
section  will  briefly  discuss  the  process  of  building  and  testing  a  hypothesis— simply 
because  it  is  one  of  the  basic  scientific  principles  of  research  methodology. 
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BUILDING  AND  TESTING  A  HYPOTHESIS 

A  "hypothesis"  is  a  statement  describing  a  relationship  between  two  or  more 
variables— a  short  statement  of  what  it  is  that  you  are  trying  to  prove.  Scientists  use 
them  as  a  way  of  keeping  their  research  within  manageable  confines. 

Of  the  two  or  more  variables  in  the  hypothesis,  one  is  the  "dependent  variable"  and 
the  others  are  "independent  variables".  The  dependent  variable  is  the  focus  of  your 
research  effort  As  a  researcher,  your  job  is  to  test  your  hypothesis  to  see  if  the 
relationship  you  described  is  indeed  true.  This  is  accomplished  by  systematically 
testing  the  independent  variables  in  order  to  measure  and  document  their  impact 
on  the  dependent  variable. 

Example 

A  researcher  wants  to  establish  if  the  use  of  a  specific  colored  background  screen  in  a 
CBT  (computer  based  training)  presentation  will  positively  affect  the  retention  of 
the  information.  So,  our  researcher  develops  the  following  hypothesis: 

The  color  of  the  background  scram  in  a  CBT  positively 
affects  the  rate  of  information  retention . 

In  this  case,  the  dependent  variable  is  the  rate  of  information  retention— this  is  what 
we  want  to  measure-and  the  independent  variables  are  die  different  possible  colors 
of  the  background  screen-these  are  what  we  will  vary  to  test  the  nature  of  the 
relationship.  In  a  hypothesis  based  experiment,  there  can  be  no  ambiguity— your  job 
is  to  test  the  hypothesis  and  ascertain  if  it  is  true,  or  not  true.  But  be  careful,  in  most 
cases  if  the  hypothesis  is  not  true,  you  do  not  necessarily  know  that  the  opposite  is 
true. 


FORMULATING  A  RESEARCH  QUESTION 

A  research  question  is  generally  more  useful  than  a  hypothesis  when  dealing  with 
issues  in  applied  research.  Building  and  testing  a  hypothesis  is  a  good  methodology 
for  solving  problems  that  begin  with  "Does...?"  and  "Can...?",  for  example.  Research 
questions,  on  the  other  hand,  are  more  effective  when  dealing  with  problems  such 
as  "Why...?"  and  "How...?". 

Formulating  a  research  question  is  much  easier  than  building  a  hypothesis,  because 
they  can  be  much  broader  in  scope. 

E&amvic 

A  researcher  wants  to  find  out  if  air  pollution  affects  an  airmans  ability  to  fly. 

So,  our  researcher  develops  the  following  research  question: 
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Does  air  quality  affect  an  airmans  ability  to  fly  ? 


nPFR  ATIONAUZING  VARIABLES 

Now  that  a  research  question  or  a  hypothesis  has  been  formulated,  the  next  step  is  to 
operationalize  all  the  variables  that  will  be  used  in  the  experiment.  This  assures  that 
there  is  no  ambiguity  as  to  what  it  is  that  you  are  studying. 

The  process  of  operationalizing  a  variable  is  quite  simple,  but  extremely  important. 
The  easiest  way  of  looking  at  it  is  that  to  operationalize  variable  "X"  you  simply 
have  to  answer  the  question:  what  do  I  mean  by  "X",  in  terms  of  this  experiment  ? 
Be  exact  f 

Technically  speaking,  there  are  two  types  of  operational  variable 
definitions-"measured"  and  "experimental1*.  Measured  definitions,  as  the  name 
implies,  describe  how  a  variable  will  be  measured  during  the  research.  An 
experimental  definition,  on  the  other  hand,  describes  how  a  concept  will  be 
represented  as  a  variable  in  die  research. 


1.01.02  Han  tin  spaclfle  naads  at  tha  training  emlremneot  la  be 
Investigated  bean  Identified? 

Every  training  environment  has  characteristics,  or  a  combination  of  characteristics 
which  are  unique  This  in  turn  influences  how  needs  can  be  addressed  within  each 
training  environment.  The  use  of  computer  based  training  by  a  highly  mobile 
squadron  could  necessitate  the  use  of  portable  computers  which  would  in  turn 
create  limitations  on  the  CBT  program,  such  as  the  use  of  color  graphics  displays. 
Even  if  a  training  related  effort  is  intended  for  Air  Force  wide  distribution  it  is 
important  to  consider  how  the  product  will  fit  in  the  training  environment 


1411.03  Have  tbs  specific  aspects  of  tbe  training  environment  to  be 
Investigated  been  Identified? 

A  research  project  may  impact  one  or  more  aspects  of  training.  The  training 
environment  is  divided  into  many  specific  areas,  each  of  which  is  important  to  the 
outcome  of  training.  These  areas  are: 

Management 

Development 

Delivery 

Evaluation 
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Documentation 


Within  each  area  are  sub  areas.  The  ISD  model  is  usually  an  appropriate  framework 
to  use  to  identify  the  specific  sub  areas  of  the  training  environment 


1 411.04  Has  a  research  strategy  been  developed? 

The  research  strategy  should  not  be  confused  with  research  design.  The  research 
strategy  describes  the  overall  technique  and  activities  that  are  required  to  assure  that 
the  research  is  carried  out.  The  factors  covered  in  the  Science  and  Technology 
checklist  can  be  used  to  provide  a  framework  of  factors  than  need  to  be  covered  in 
order  to  carry  out  a  successful  research  effort 

Some  of  the  important  factors  to  consider  in  a  research  strategy  are: 

o  address  recognized,  real,  and  substantial  needs 
o  with  realistic  objectives 
o  using  feasible  enabling  technologies 

o  have  customer  participation  and  high-level  customer  support 
o  in  a  risk-tolerant  research  and  development  environment 
o  have  competent  people 
o  have  quality  leadership 

Without  a  strategy  carefully  laid  out,  the  likelihood  of  canying  out  a  successful 
research  effort  is  limited. 


1.01.04.01  Havo  research  methods,  solutions,  or  options  been  selected  to 
satisfy  tbe  requirements? 

Practically  any  research  problem  can  be  addressed  using  a  variety  of  different 
research  methods.  However,  the  most  appropriate  research  methodology  is  one  that 
takes  advantage  of  environmental  factors  such  that  it  would  produce  the  greatest 
gains  for  the  least  expense.  A  problem  that  at  first  glance  is  a  purely  an  experimental 
or  developmental  effort  could  be  enhanced  by  conducting  surveys  and  interviews  if 
their  are  indications  that  the  solution  to  the  problem  may  already  exist,  in  bits  and 
pieces.  In  such  a  case  gathering  existing  findings  can  reduce  much  of  the 
experimental  work  needed  to  produce  an  application. 

At  other  times  survey  research  may  appear  to  be  the  best  approach.  The  research 
should  first  determine  if  the  information  can  be  gathered  from  existing 
documentation.  The  basic  idea  is  "not  to  re-invent  the  wheel." 


1.01.04.02  Have  assumptions  been  developed,  documented,  and  coordinated? 

For  practically  any  research  effort,  the  researcher  must  of  necessity  make  certain 
assumptions.  These  assumptions  may  deal  with  such  issues  as  the  availability  of  or 
access  to  information  and  personnel,  the  cooperation  of  outside  organizations,  the 
available  facilities,  and  "no  cost”  support  for  certain  activities,  such  as  scheduling 
research  participants  or  survey  respondents. 

The  researcher  should  not  leave  these  assumptions  as  assumptions,  but  actively 
strive  to  determine  if  what  is  assumed  is,  in  fact,  correct  If  support  for  example,  is 
expected  from  a  MAJCOM,  contact  the  MAJCOM  and  assure  that  the  support  will  be 
forthcoming.  At  times  the  researcher  may  find  that  support  promised  was  only 
valid  within  a  very  specific  time  frame.  To  avoid  problems  resulting  from 
unsupported  assumptions  the  researcher  should  carry  out  die  following  activities. 

a.  List  all  assumptions  or  expectations  that  are  felt  necessary  in  order  for 
the  project  to  be  successful. 

b.  Check  to  determine  if  such  assumptions  are  correct  and  document  the 
findings. 

c  If  a  critical  assumption  does  not  prove  to  be  correct  determine  if  the 
assumption  was  absolutely  essential  or  that  the  problem  that  arises  can 
be  addressed  in  other  ways.  It  may  prove  necessary  to  reconsider  the 
feasibility  of  the  research. 


1JI1.04.03  Hava  problems  In  Ibo  research  approach  been  Idenflfled? 

Identifying  potential  problems  in  a  research  activity  and  planning  to  deal  with  them 
can  often  eliminate  or  lesson  the  effect  of  the  problem  on  the  research.  There  are 
two  classes  of  problems  related  to  any  research  approach: 

a.  Problems  inherent  to  the  particular  research  approach.  Every  research 
approach,  whether  it  be  experimental,  survey,  archival,  or  any  method 
or  combinations  of  methods,  will  have  inherent  flaws.  Numerous 
volumes  have  been  written  on  the  subject  and  most  researchers  have 
taken  course  work  that  at  least  in  part  address  inherent  weaknesses  in  a 
research  approach. 

b.  Problems  unique  to  the  application  of  a  research  approach  in  a  specific 
environment.  In  some  cases  a  research  approach  may  have  proven 
very  successful  in  one  environment  and  a  clear  failure  in  only  a 


slightly  different  environment.  The  researcher  must  ask  what  would 
cause  a  specific  approach  not  to  suoceed. 


1.01.05  Has  l he  research  Identified  specific  areas  or  problems  where 
appllcatloo  of  technoloov  may  be  appropriate? 

The  application  of  technology  to  a  problem  can  often  prove  to  be  the  most  cost- 
effective  and  useful  method  of  addressing  an  issue.  Yet  at  other  times  it  may  in  fact 
be  totally  non-productive.  Care  must  be  taken  to  realistically  appraise  the  current 
situation  that  is  being  analyzed  to  make  a  cost-benefit  based  decision  as  to  the 
relevancy  and  utility  of  a  technologically  based  solution. 


1.01.08  Hr  a  plan  of  action  been  developed? 

A  plan  of  action  is  a  written  description  on  how  to  carry  out  the  proposed  research. 
The  plan  of  action  should  not  only  list  what  has  to  be  done,  but  also  how  it  is  to  be 
done,  and  in  what  order  the  activities  will  be  performed.  A  detailed  plan  of  action, 
regardless  of  its  format,  needs  to  be  as  specific  as  possible.  The  plan  of  action  should 
be  a  map  of  all  the  activities  that  need  to  be  performed  to  get  from  Problem  to 
Solution. 

The  plan  of  action  may  have  to  be  re-visited  and  modified  during  the  life  of  a 
research  project  Often  the  solution  to  a  problem  is  not  absolutely  dear  but  as  the 
research  progresses  ft  becomes  more  focused. 


1.01.07  Will  the  findings  of  Oic  plan  of  action  lead  to  technology  required? 

If  the  desired  solution  is  technology  based,  it  is  important  that  the  plan  lead  to  a 
technological  solution,  IF  FEASIBLE  Any  researcher  must  be  willing  to  accept  the 
possibility  that  the  initial  concept,  and  the  assumption  that  technology  is  the 
appropriate  solution  may  not  be  correct  For  some  research  activities  the  need  for 
technology  is  obvious  and  documented.  In  such  cases  the  researcher  should  consider 
whether  the  technology  being  considered  is  appropriate.  Technologies  that  can 
address  a  problem  —  even  related  technologies  —  can  differ  enough  to  cause 
significant  differences  in  outcomes.  An  expert  system  job  aid  built  around  an  AI 
language  such  as  Pro  Log  would  take  less  time  to  develop  cost  less,  and  be  more 
reliable  than  if  the  research  effort  required  that  all  programming  be  done  in  ADA 
where  the  basic  shell  would  have  to  be  created  from  scratch. 


1.02 


Characteristics  of  the  Protlucts(s)  to  be  Produced: 


1.02.01  Have  the  physical  characteristics  of  the  product(s)  to  be 
produced  been  Identified? 

Whatever  is  produced  must  fit  in  the  environment  in  which  it  will  be  used.  Even  a 
printed  document  must  be  structured  appropriately  for  use.  The  approximate 
format  and  language  must  fit  the  user.  A  document  intended  for  use  by  research 
may  have  little  resemblance  in  the  language  and  structure  of  the  report  from  one 
intended  for  decision  makers. 

For  any  physical  product,  the  physical  characteristics  can  be  significant  If  the 
product  is  hardware,  or  partly  hardware,  its  size  and  weight  may  be  an  important 
consideration. 


1.02.02  Hava  tfao  functional  characteristics  of  the  productfs)  to  be 
produced  been  Identified? 

The  functional  characteristics  of  a  product  should  be  based  on  its  intended  use  and 
the  characteristics  of  the  intended  users.  Ignoring  either  factor  can  result  in  a  poor 
product.  If  the  product  is  software,  the  programming  style  (for  example,  block 
programming  for  ease  of  modification)  and  user  interface  screen  design  are 
important  characteristics. 


1.03  Externally  Oenerated  Priorities: 

1.03.01  Have  priorities  for  research  been  established  by  "external" 
activities  (e.o.  Air  Staff,  MAJC0M,  user)? 

■Hie  researcher  must  consider  the  potential  user's  needs.  Without  a  "customer"  the 
likelihood  of  a  research  product  actually  becoming  operational  is  reduced 
significantly.  External  support  strengthens  the  visibility  and  viability  of  a  research 
effort.  An  external  potential  user  or  party  interested  in  the  outcomes  of  a  research 
effort  should  be  directly  involved  in  setting  goals  for  the  research  activity.  This  does 
not  mean  that  an  external  party  needs  to  be  involved  in  every  phase  of  the  research 
design  and  execution  only  that  their  needs  be  considered.  It  would  make  little  sense 
to  develop  the  most  advanced  training  tool  in  the  world  if  it  was  so  manpower  and 
capital  intensive  as  to  eliminate  even  the  consideration  of  the  tool  being  used.  At 
least  some  potential  customers  should  be  involved,  and  their  concerns  should  be 
addressed. 
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1.0101.01  Have  the  above  priorities  been  accommodated  by  the  research 
plan  and  approach? 


This  is  a  Yes  /  No  question. 

1.0101.02  If  the  above  priorities  have  not  been  accommodated,  has  this 
been  discussed  with  the  official  who  established  the  priorities? 

This  is  a  Yes  /  No  question. 

1J3.01.03  Does  that  official  agree  with  your  "re-prioritization?11 

If  the  answer  is  NO  the  researcher  should  be  looking  for  other  customers,  or  trying 
to  change  the  mind  of  those  who  disagree  with  the  re-prioritization. 

1.0102  At  least  annually,  have  you  checked  with  these  external  activities 
to  reconfirm  their  priorities? 

If  possible,  a  formal  briefing  would  be  the  best  method  to  assure  continued  support, 
especially  when  a  significant  staff  or  leadership  change  occurs.  Priorities  change 
over  time  and  between  individuals.  Changing  conditions  could  have  reduced  the 
importance,  and  funding,  or  your  specific  research.  Involving  the  customer  in 
setting  priorities,  and  then  checking  with  them  to  reconfirm  priorities  is  absolutely 
essential.  Not  involving  the  customer  is  a  critical  shortcoming  and  should  be 
addressed. 

1.04  Internally  Generated  Priorities: 

1.04.01  Is  there  clear  rational  for  internally  generated  priorities? 

In  most  cases  research  efforts  are  not  carried  out  by  a  laboratory  to  satisfy  a 
"customer's"  needs.  A  research  effort  may  supplement  other  ongoing  research  or 
provide  information  needed  for  additional  research.  The  research  organizati 
may,  and  most  likely  does  have,  its  own  research  agenda,  and  a  specific  research 
activity  may  have  been  developed  primarily  because  it  advances  that  research 
agenda. 

1.04.02  Are  the  internally  generated  priorities  compatible  with  the 
externally  generated  priorities? 

Yes  /  No.  If  no  this  issue  must  be  faced  and  a  solution  developed. 


1.04.03  Is  Own  external  otersioM  lor  Internally  generated  priorities? 


A  research  project  can  often  benefit  from  the  presence  of  external  oversight  for  some 
of  its  internal  activities.  The  presence  of  another  perspective  can  often  point 
broaden  the  scope  of  options  available  to  the  research  team. 

If  there  is  external  oversight,  a  procedure  for  consultation  and  appraisal  should  be 
set  in  place  at  the  outset  of  the  project  to  ensure  that  there  is  a  sense  of  cooperative 
effort  in  the  project  rather  than  an  occasional  intrusive  presence. 


1  MM  Co  ( will)  i  m  agency  commit  ta  t&a  pralact  aatf  Its  axa  acted  nme-lliie? 

This  particular  question  may  well  be  revisited  time,  after  time,  by  a  researcher  till 
the  agency  commits  to  a  project  Since  the  competition  within  a  research  laboratory 
for  limited  funding  is  high,  worthwhile  efforts  are  often  rejected.  Decision  makers 
have  to  make  the  decisions  on  various  grounds.  If,  for  example,  a  highly 
worthwhile  research  proposal  is  submitted,  but  its  funding  requirements  are  such 
that  several  other  equally  important  projects  would  have  to  be  dropped,  decision 
makers  may  have  little  choice  and  not  fund  the  more  expensive  effort 

If  a  project  is  rejected  the  researcher  should  find  out  why.  With  information,  it  may 
be  possible  that,  in  die  next  funding  cycle,  a  stronger  case  can  be  made,  or  the  project 
can  be  restructured  so  as  to  have  a  better  fit  with  the  overall  research  agenda,  or  with 
the  funding  available. 
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2.  THE  TECHNOLOGIES 


2.01  Identification  of  Technologies: 


Technology"  can  be  defined  as  the  process  of  applying  scientific  knowledge  to  any 
activity  or  operation  in  order  to  increase  its  efficiency.  Using  technology  in 
problem-solving  requires  a  systematic  approach,  and  must  take  into  consideration 
many  factors. 

Technology  is  often  mistakenly  defined  only  in  terms  of  the  hardware  associated 
with  the  technological  revolution:  computers,  missiles  and  other  "physical"  objects. 
However,  in  truth,  technology  encompasses  not  only  such  hardware,  but  ideas  and 
concepts  as  well. 

In  a  broad  sense,  technology  can  be  considered  as  having  two  main  components: 

(1)  a  hardware  aspect,  consisting  of  the  tool  diet  physically  embodies  the 
technology. 

(2)  a  software  aspect,  consisting  of  the  information  base  for  the  tool 

Although  the  interdependence  between  a  tool  and  the  way  in  which  it  is  used,  is 
critical  to  accomplishing  a  task,  the  tool  is  easier  to  identify  with.  It  is  because  of  this, 
that  people  often  think  of  technology  mainly  in  terms  of  hardware.  Understanding 
this  interaction  or  interdependence  is  the  key  to  the  successful  use  of  technology. 

In  a  general  sense,  in  many  cases  the  hardware  side  of  a  technology  is  dominant, 
mostly  in  cases  where  the  user  is  not  a  significant  part  of  die  process.  But  in  other 
cases,  a  technology  may  be  almost  entirely  comprised  of  information,  ideas  and 
concepts. 


2.01.01  Hate  toe  characteristics  of  the  needed  technologies  been 
Identified? 

An  analyst  should  take  a  proactive  approach  to  identifying  the  characteristics  of  a 
technology  before  making  any  attempt  to  select  a  technology  to  fulfill  a  need.  A 
"proactive  approach"  involves  defining  the  characteristics  of  the  needed 
technologies.  By  carefully  defining  the  needed  characteristics,  the  analyst  can  avoid 
the  two  greatest  pitfalls  of  new  technology  implementation:  (1)  investing  so  much 
that  system  capabilities  are  neither  utilized  nor  cost-effident,  or,  (2)  investing  so 
little  that  the  system  becomes  an  obstade  rather  than  an  enhancement  to 
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productivity.  The  problem  of  finding  the  correct  balance  for  specific  requirements 
can  be  difficult  to  surmount.  Finding  the  right  balance  before  any  investments  are 
made  is  a  critical  part  of  this  process. 


The  first  step  taken  by  the  analyst  to  perform  this  evaluation  should  be  to  formally 
identify  the  need  for  a  new  technology  and  to  describe  its  characteristics.  This  could 
indude,  for  example 

•  Determining  the  desirability  and  practicality  of  automating  a  manual 
system, 

•  Looking  for  ways  to  make  existing  processing  capabilities  more  efficient  and 
responsive,  or 

•  Attempting  to  enhance  an  existing  automated  system  by  adding 
processing  capabilities  to  meet  new  requirements. 


The  identification  of  the  characteristics  of  the  needed  technology  can  be 
accomplished  by  the  following  key  activities: 


Identify  and  define  assumptions. 

Assumptions  are  simply  statements  of  die  organizational  and  managerial 
priorities,  and  the  technical  and  operational  considerations  which  define  the 
contract  of  the  proposed  project 


One  of  the  first  objectives  is  to  identify  and  define  the  assumptions  that  affect  the 
scope  and  validity  of  the  need. 


Establish  evaluation  criteria. 

In  any  type  of  analysis,  solid  evaluation  criteria  are  necessary  in  order  to 
properly  select  a  solution  which  truly  meets  the  needs  of  the  organization. 
This  criteria  will  be  useful  in  evaluating  alternative  solutions  which  may  be, 
or  may  already  have  been  proposed  in  the  attempt  to  meet  the  expressed 
need. 

While  most  often  this  criteria  will  be  used  to  "maximize  net  benefits"  (e.g.,  to 
identify  the  solution  whose  measurable  benefits  most  exceed  its  measurable 


costs),  other  criteria  which  can  be  considered  maybe  useful  in  minimizing 
costs  or  for  obtaining  the  best  cost/benefit  ratio. 


Identify  and  define  alternative  solutions. 

It  is  recommended  that  at  least  two  distinct  alternative  solutions  be  generated. 
In  some  situations,  this  may  be  accomplished  by  simply  varying  such 
characteristics  as  timeliness,  flexibility  or  security. 


Identify  cost/benefits. 

This  activity  has  three  parts,  as  follows: 

(1)  Identify  costs. 

All  expected  areas  should  be  determined  where  costs  may  be  incurred. 
Areas  of  possible  costs  that  should  be  reviewed  are  procurement, 
start-up,  project  related,  operating,  security,  etc 

(2)  Identify  benefits. 

All  types  of  benefits  that  art'  expected  to  accrue  should  be  determined. 
Potential  benefits  include  reduction  in  per  unit  costs,  improved 
accuracy,  increased  speed  of  operation,  etc 

(3)  Adjust  for  present  value. 

After  identifying  the  specific  cost  and  benefits  for  each  alternative 
solution  for  the  need,  cost  figures  should  be  adjusted  for  their  present 
value. 


Analyze  results. 

If  two  or  more  alternatives  solutions  appear  feasible  and  have  similar  total 
net  benefits,  the  critical  assumptions  that  affected  the  results  should  be 
re-examined. 

Develop  project  development  plan. 

The  project  development  plan  is  evolutionary.  Plans  and  schedules  are 
continually  adjusted  as  analyses  result  in  more  definitive  assessments  and 
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projections.  Milestones  are  scheduled  and  major  tasks  are  reduced  to  smaller, 
more  discrete  tasks. 


2JI1.02  Was  a  market  survey  performed  to  Identify  viable  commercial 
products  that  best  satisfy  this  requirement? 

To  discover  whether  or  not  a  market  survey  was  performed  to  identify  commercial 
products  that  might  exist  to  satisfy  this  requirement,  the  analyst  should  be  aware  of 
the  following: 


•  Problem  definition 

A  concise  statement  of  the  topic  in  question  is  very  important  Without  a  precise 
definition  of  the  topic,  the  analyst  may  collect  irrelevant  and  expensive  information 
and  confuse  rather  than  clarify  issues.  A  good  definition  of  the  problem  directs  the 
analyst  toward  the  collection  and  analysis  of  specific  information  to  satisfy  his/her 
requirement 


•  Sources 

Market  survey  information  is  available  from  both  government  and 
non -government  sources.  Three  major  sources  of  commercial  or  non-government 
information  are: 


Regularly  published  periodicals  contain  articles  on  various  aspects  of 
marketing  surveys  and  are  available  in  most  libraries.  Some  are  quite 
broad  in  scope  (Business  Week.  Toumal  of  Marketing);  others  are  more 
specialized  (journal  of  Advertising,  journal  of  Consumer  Research). 
These  periodicals  are  published  by  professional  associations,  regular 
publishing  companies  or  trade  associations.  The  orientation  of  these 
periodicals  varies  widely.  Selected  examples  are: 


weekly 


quarterly 


bimonthly 


quarterly 


weekly 


quarterly 
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Chain  Store  Age,  monthly 

Columbia  Toumal  of  World 
Basmsaa,  quarterly 

Editor  it  Publisher's  Market 
Guide,  annual 

Fortune,  semimonthly 

Graphic  Guide  to  Consumer 
Markets,  annual 


Toumal  of  Small  Business 
Management,  quarterly 

MSU  Business  Topics,  quarterly 

Nielsen  Researcher,  bimonthly 


ve  Grocer,  monthly 


Rand  McNally  Commercial  Atlas 
it  Marketing  Guide,  annual 


Harvard  Business  Review, 
bimonthly 


Sales  &  Marketing  Management. 
monthly 


Toumal  of  the  A 
Marketing  Science, 
quarterly 


•r-cwii 


V  of 


Toumal  of  Advertising. 
bimonthly 


Stores,  monthly 


University  of  Michigan 
Business  Review, 
quarterly 


Tpumaj  of. Advgrfejng 
Research,  quarterly 


Books,  monographs,  and  other  nonrecurring  literature  are  published  by  a  number  of 
organizations.  Some  groups,  such  as  the  American  Marketing  Association, 
disseminate  information  to  increase  knowledge  and  professionalism.  Others,  such 
as  the  Better  Business  Bureau,  are  involved  in  self-regulation  and  public  opinion. 
Yet  another  type,  such  as  the  National  Retail  Merchants  Association,  functions  as  a 
spokes  person  for  an  industry  as  well  as  an  information  disseminator.  Each  of  these 
organizations  distributes  materials  for  a  nominal  fee  or  free  of  charge.  The  following 
publish  a  wide  variety  of  information  pertaining  to  marketing: 

American  Management  Association  Banks 
American  Marketing  Association  Foundations 
Better  Business  Bureau  Research  institutes 
Chamber  of  Commerce  Trade  associations 
Other  Universities 
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Commercial  research  houses  conduct  periodic  and  ongoing  studies  and  mike  the 
results  of  the  studies  available  to  ma  \y  clients  for  a  fee.  The  fee  can  be  quite  low  or 
range  into  the  tens  of  thousands  of  dollais,  depending  upon  the  extent  of  the 
information.  The  following  commercial  research  houses  sell  information  to 
subscribers: 


A.C.  Nielsen 
Arbitron 

Audit  Bureau  of  Circulation 
Audits  and  Surveys 
Dun  and  Bradstreet 
Market  Research  Corporation 
of  America 


National  Family  Opinion 
National  Purchase  Diary  Panel 
R.L.  Polk  (mailing  lists, 
automobile  data) 

Selling  Area-Marketing,  Inc 
(SAMO 

Standard  Rate  and  Data  Service 


*  Advantages 

This  type  of  information  is  inexpensive,  easy  to  assemble,  and  provides  background 
data  about  the  problem.  There  frequently  are  several  sources  of  secondary  data. 
These  allow  the  analyst  to  obtain  various  perspectives,  to  gather  large  amounts  of 
information  and  to  verify  data.  When  information  is  assembled  by  an  independent 
source  such  as  Fortune  or  Business  Week,  the  results  are  believable.  Both  of  these 
sources  have  a  high  level  of  credibility  and  a  reputation  for  thoroughness. 

*  Disatiyantagea 

Available  information  may  not  suit  the  purposes  of  current  research  because  it  was 
collected  for  other  reasons.  For  example,  the  units  of  measure  may  be  different  from 
what  are  needed.  The  information  may  be  dated  or  obsolete.  It  may  have  outlived  its 
usefulness.  The  precision  with  which  the  information  was  collected  may  be  lacking. 
The  purpose,  data  collection  technique,  and  method  of  analysis  of  the  original  study 
should  each  be  examined  for  bias.  Lastly,  the  reliability  of  the  information  is  not 
always  known.  Many  research  projects  are  not  re- tested 


2.01.03  Has  a  literature  search  been  conducted  to  determine  what 
technological  capabilities  are  available  tram  Industry? 

Although  military  R  St  D  is  often  at  the  forefront  of  cutting  edge  research,  there  may 
be  parallel  developments  taking  place  in  the  private  sector.  In  order  to  save  costs 
and  avoid  replication,  conducting  a  literature  search  to  determine  what 
technological  capabilities  are  available  from  industry  is  almost  always  a  good  idea.  If 


the  literature  search  does  not  yield  a  solution,  it  will  at  least  provide  guidelines  to 
determine  the  direction  of  the  existing  research. 


There  are  many  resources  and  tools  available  to  conduct  literature  searches.  The 
most  widely  used  are  a  variety  of  on-line  data  bases  such  as  the  selected  listing 
provided  below: 


AB1/INFORM  (Abstracted  Business  Information) 

1971  to  present,  updated  monthly 

Covers  550  international  business  management  publications  in  such  Helds  as 
economics,  accounting,  marketing,  management  science,  insurance  and  real 
estate. 


AIM/ ARM  (Vocational  and  Technical  Edu 


A  specialized  index  of  information  about  vocational  and  technical  education 
and  related  areas.  It  covers  materials  published  between  1967  and  1976.  More 
recent  publications  have  been  incorporated  into  the  Educational  Resources 
Information  Center  (ERIC). 


BOOKS  INFO 

Contains  citations  of  books  currently  in  print  Provides  information  about  the 
author,  publisher,  price,  subject  descriptors,  LC  card  number,  and  ISBN 
number. 


COMPREHENSIVE  DISSERTATION  INDEX 

1961- to  gssfinL-iiEdatfid  monthly 

Includes  most  dissertations  in  the  United  States  and  a  selected  listing  from 
foreign  universities.  Citations  are  provided  so  that  abstracts  may  be  found  in 
the  published  Comprehensive  Dissertation  Index  volumes.  Complete  copies 
of  the  dissertations  may  be  obtained  from  University  Microfilms,  Ann  Arbor, 
Michigan. 

Congressional  Information  Service 
1970  to  present,  updated  monthly 

A  guide  to  the  publications  of  the  United  States  Congress  that  cover  hearings, 
committee  reports,  and  s pedal  committee  and  subcommittee  publications. 

ERIC  (Educational  Resources  Information  Center) 
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1966  to  present,  updated  mi 

Contains  indexes  of  over  700  periodicals  and  several  hundred  thousands  of 
research  reports,  projects,  and  monographs  in  education. 


THE  INFORMATION.. BANK 

1974  to  present,  updated  monthly 


Compiled  by  the  New  York  Times  and  other  major  newspapers,  and  provides 
abstracts  of  articles  on  current  events  from  over  60  newspapers,  magazines, 
scientific  and  financial  periodicals. 


INFORMATION  INDUSTRY,  MARKETPLACE  1981:  An  International 
Directory  of  Information  Products. and -Services 

A  comprehensive  guide  to  companies  and  organizations  providing  computer 
products,  supplies  and  services.  The  publication  lists  data  base  publishers, 
machine  readable  data  bases,  printed  products  derived  from  data  bases,  and 
support  services  and  suppliers. 


INSPEC  (Fhvsics.  Electronics.  C 
1969  to  present  updated  monthly 


Indudes  physics,  electrical  and  electronic,  and  computer  and  control  abstracts. 


NATIONAL  NEWSPAPER-INDEX 
I2Z2JO.  present.  updated  monthly 

includes  the  Chrigtten  Sfitenre  Monitor,  New  York  limes.  Wall  Siregt 
Ig.um?L  Los  Angeles  Tunes  and  the  Washington  Post 

EES 

A  guide  to  the  tJ.S.  and  international  manufacturing,  marketing  and 
industrial  forecasts. 

SCISEARCH 

to  present,  updated  semi-monthlv 

A  multi  disdpUnary  index  which  provides  access  to  90  per  cent  of  the  world's 
significant  sdentitic  and  technical  literature.  It  contains  references  from  both 
the  S£jgncs.Citetign.  Index  and  the  Current  Contents. 


1963  to  present,  updated  monthly 

Contains  an  index  of  1000  social  science  journals  and  social  science  articles 
from  over  2200  other  journals. 

SOCIOLOGICAL  ABSTRACTS 

1963  to  present  updated  quarterly 

Contains  an  index  of  world  literature  on  sociology  and  related  disciplines.  It 
includes  selected  articles  from  over  1200  journals. 

SSIE  (Smithsonian  Science  Information  Exchange) 

Contains  reports  of  government  and  privately  funded  science  and  social 
science  research  projects,  either  in  progress  or  initiated  and  completed,  during 
the  past  two  years.  It  provides  the  name  of  the  project  investigator,  funding 
agency,  summary  and  detailed  subject  descriptors. 

There  are  also  a  wide  variety  of  published  directories  and  guides  to  secondary 
sources  of  research  information.  A  listing  of  selected  sources  is  provided 
below: 

Applied  Science  and  Technology  Index 

A  subject  index  to  over  200  journals  in  aeronautics,  space  science,  energy 
engineering,  fire  prevention,  chemistry,  computer  technology,  food  and  food 
industries,  geology,  machinery,  mathematics,  mineralogy,  metallurgy, 
oceanog  <phy,  petroleum  and  gas,  physics,  textiles  and  fabrics,  and 
transportation. 

Communication  Abstracts 

A  quarterly  index  of  major  articles,  books,  and  monographs  related  to 
communication.  It  contains  abstracts  of  indexed  publications. 

The  Directory  of  Directories 

An  annual  compilation  of  directories.  It  is  a  "road  map"  to  guides  indexed  by 
subject  matter  and  title. 

Dissertation  Abstracts  International 

A  publication  including  titles,  key  words  and  author  indices  for  doctoral 
dissertations  in  over  350  institutions  in  the  United  States  and  abroad.  It 
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provides  dissertation  abstracts  and  is  published  monthly.  Complete  copies  of 
dissertations  can  be  ordered  from  University  Microfilms  of  Ann  Arbor, 
Michigan. 


Enon«ring.lnto 

A  monthly  guide  to  engineering  literature. 


Human  Resources  Abstracts 
(formerly  Poverty  and  Human  Resources) 


An  abstract  journal  of  Sage  Publications  providing  coverage  of  human 
resource  and  social  problems  ranging  from  slum  rehabilitation  and  job 
development  training  to  compensatory  education,  minority  group  problems 
and  rural  poverty. 

INFORM 

Contains  abstracts  of  articles  in  over  400  English-language  management  and 
administrative  science  journals. 


New  York  Times  Index 


A  guide  to  the  contents  of  the  New  York  Times  and  includes  brief  summaries 
of  articles. 


Readers  Guide  to  PmodiaLLitgutua 


A  directory  containing  subject  and  author  indexes  of  more  than  150  general 
publications  in  the  United  States.  It  has  been  published  semi-monthly  since 
1900. 


Science  Cit 


l  Index 


Comparable  to  the  SSCI  (see  below)  but  for  the  natural  sciences.  It  indudes 
approximately  90  per  cent  of  the  significant  sdentific  and  technical  literature 
published  worldwide. 


Social  Science  Citation  Index  (SSCI) 

Indexes  the  significant  items  from  approximately  1000  worldwide  sodal  and 
behavioral  science  journals  in  the  natural,  physical,  and  biomedical  sdenoes. 
It  indudes  citation  index,  author  index  (first  author  only),  and  subject  index. 
It  has  been  published  by  the  Institute  for  Sdentific  Information  since  1969. 
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Social  Science  Index 

A  subject  and  author  index  to  articles  in  more  than  260  journals  in  the  social 
and  behavioral  sciences,  law,  medicine,  and  related  subjects. 

Sociological  Abstracts 

Provides  abstracts  of  selected  articles  from  over  1200  articles  in  sociology  and 
related  disciplines. 

Trade  and  Industry  Index 

An  index  that  provides  complete  listing  and  selective  abstracts  of  275  trade 
and  industry  journals  and  selective  indexing  of  1200  additional  publications. 

Wall  Street  loumal  Index 

A  guide  to  the  contents  of  the  Wall  Street  Journal.  It  contains  two  separate 
published  sections  covering  general  news  and  corporate  news. 

Work  Related  Abstracts 

An  annotated  index  of  books,  articles,  and  dissertations  covering  labor 
relations,  personnel  management,  and  organizational  behavior. 

(This  list  of  on-line  data  bases  and  secondary  research  guides  was  compiled  from 
Secondary  Research,  D.W.  Stewart,  Sage  Publications,  1984). 


2.01.04  Can  existing  technology  be  modified  or  adapted  to  satisfy  tills 
requirement? 

Before  embarking  upon  the  development  or  acquisition  of  a  new  technology  to 
satisfy  a  determined  need,  it  is  a  worthwhile  exercise  to  investigate  existing 
technology  to  determine  if  it  can  be  modified  or  adapted  in  some  way  to  satisfy  the 
requirement 

One  technique  that  may  be  used  to  investigate  this  possibility  is  to  analyze  the  ways 
in  which  the  workers  are  currently  using  the  existing  technology  to  meet  this  new 
requirement.  Despite  the  stringent  development  and  implementation  guidelines  of 
researchers,  the  average  worker  will  often  find  a  way  to  "re-invent"  existing 
technologies  to  solve  the  new  problem.  Therefore,  by  analyzing  the  ways  in  which 
the  workers  are  using  existing  technology  in  a  manner  other  than  intended,  the 
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analyst  can  often  gain  valuable  insights  into  how  an  existing  technology  can  be 
formally  modified  or  adapted  to  fit  a  new  requirement 

In  addition,  the  compatibility  of  a  new  technology,  or  idea,  with  existing  technology 
can  either  accelerate  or  retard  the  rate  of  adoption.  It  is  often  hard  for  end-users  to 
deal  with  a  new  technology  except  on  the  basis  of  the  familiar  and  old  fashioned. 
Previous  pracdoe  is  a  familiar  standard  against  which  the  new  technology  can,  and 
most  likely  will,  be  interpreted,  as  users  assess  the  new  mol,  system  or  method.  So, 
the  analyst  must  strongly  consider  that  the  rate  of  adoption  of  new  technology  is 
affected  by  the  existing  idea  that  it  supersedes. 

It  is  obvious  that  if  new  technology  were  completely  congruent'  with  existing 
practice,  there  would  be  no  need  for  innovation.  In  other  words,  the  more 
compatible  new  technology  is  with  the  old  way  of  doing  something,  the  less  of  a 
change  it  represents.  So  what  is  the  use,  then,  of  the  introduction  of  a  very  highly 
compatible  technology?  The  introduction  of  a  compatible  technology  is  useful  as  a 
first  step  in  a  series  of  innovations  that  are  to  he  introduced  sequentially.  The 
compatible  technology  paves  the  way  for  later,  less  compatible  innovations. 


2.01.04.01  Was  a  market  survey  performed  to  Identify  viable 
government-owned  products  that  best  satisfy  Ibis  requirement? 

To  discover  whether  or  not  a  market  survey  was  performed  to  identify  viable 
government-owned  products  that  might  exist  to  satisfy  this  requirement,  use  the 
concepts  discussed  above  for  the  commercial  sector.  They  hold  true  for  the 
government  as  well. 


In  searching  for  this  information  it  is  important  to  note  that  the  government  itself 
is  usually  the  best  source  for  information  about  its  activities.  The  government 
routinely  collects  and  distributes  a  wide  range  of  statistics  and  descriptive  materials. 
Government  agencies  publish  many  pamphlets  on  topics  such  as  franchising, 
pricing,  credit,  product  warranties,  and  deceptive  sales  practices.  These  materials  are 
usually  distributed  free  of  charge,  or  sold  for  a  nominal  fee.  The  Monthly  Catalog  of 
United  States  Government  Publications  contains  a  listing  of  these  items. 


Sources  -  Market  survey  information  of  government-owned  products 
is  available  from  two  major  sources: 


Selected  publications  are: 

American  Statistical  Index  (Congressional  Information  Service), 
annual  with  monthly  updates 
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Annual  Survey  of  Manufacturers  (Department  of  Commerce), 
annual 

Business  Statistics  (Office  of  Business  Economics),  biennial 

Catalog  of  U.S.  Census  Publications  (Bureau  of  Census),  quarterly 

Census  of  Manufacturers  (Bureau  of  Census),  every  five  years 
ending  in  2  and  7 

Census  of  Population  (Bureau  of  Census),  every  ten  years  ending 
in  0 


Census  of  Retail  Trade,  Wholesale  Trade,  and  Selected  Service 
Industries  (Bureau  of  Census),  e  very  five  years  ending  in  2  and  7 


Census  of  Tra 
ending  in  2  and  7 


(Bureau  of  Census),  every  five  years 


Countv  and  Cltv  Data  Book  (Department  of  Commerce),  several 
times  each  decade 

Federal  Reserve  Bulletin  (Federal  Reserve  System),  monthly 

Monthly  Labor  Review  (Bureau  of  Labor  Statistics),  monthly 

Monthly  Urban  Review  (Bureau  of  Labor  Statistics),  monthly 

Statistical  Abstract  of  the  United  States  (Department  of 
Commerce),  annual 


Survey  of  Current  Business  (Office  of  Business  Economics), 
monthly 

Vital  Statistics  Report  (Health,  Education,  and  Welfare), 
monthly 


Government  agencies  of  interest: 

Bureau  of  Census  National  Center  for 

Bureau  of  Labor  Statistics  Educational  Statistics 

Congressional  Information  National  Center  for  Health 
Service  Statistics 
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Consumer  Product  Safety  National  Technical  Information 
Commission  Service 

Department  of  Commerce  Office  of  Business  Economics 
Department  of  Labor  Small  Business  Administration 

Environmental  Protection  Statistical  Reporting  Service, 
Agency  Department  of  Agriculture 

Federal  Reserve  System  United  States  Postal  System 

Federal  Trade  Commission 
Food  and  Drug  Administration 


2.01.04.02  Has  tha  Investigation  of  currently  available  technology  produced 
results? 

If  the  answer  to  this  question  is  yes,  the  analyst  must  determine  whether  it  is 
possible  to  modify  the  available  technology  to  fulfill  die  purpose  of  his/her  project. 
This  determination  should  be  made  incansideration  of  such  issues  as  survivability 
in  combat,  lifespan  of  equipment,  compatibility  with  existing  equipment,  cost 
effectiveness,  etc 


2.01.05  Has  a  literature  search  been  conducted  to  determine  what 
technological  capabilities  are  forecasted  by  Industry? 

Even  if  the  required  technology  is  outside  the  scope  of  current  private  sector 
research,  it  is  worthwhile  to  monitor  the  trends  of  any  related  industries.  This  will 
serve  two  purposes.  First,  it  will  prevent  investment  in  an  area  in  which  a  major 
research  has  already  taken  place,  and  positive  results  are  soon  expected.  Secondly,  it 
will  ensure  that  any  new  technology  that  is  developed  can  be  designed  in  such  a  way 
that  it  can  conform  to  any  newly  created  or  developing  standards. 

Equipment  designed  to  be  flexible  and  easily  modified  to  satisfy  new  requirements 
has  a  higher  return  on  investment  than  equipment  that  cannot  adapt  to  new 
circumstances. 

(See  paragraph  2.01.03  for  a  selected  listing  of  on-line  information  data  bases 
that  may  aid  in  this  investigation.) 


2.01.06  Has  tbo  Investigation  of  technology  In  development  produced 
promising  results? 

Technological  innovation  implies  the  introduction  of  something  new  —  a  new  idea, 
method,  or  device.  The  analyst  must  be  able  to  recognize  whether  or  not  something 
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truly  new  and  creative  has  been  developed,  or  If  promising  results  have  been 
produced,  through  the  investigation  of  current  technological  development.  To 
address  this  issue  the  analyst  must  understand  the  role  technological  innovation 
plays  in  the  research  process. 


2.01.07  Are  now  technologies  required  to  address  the  requirement? 

At  this  stage,  the  analyst  must  decide  if  new  technologies  are  required  to  address  the 
requirements.  If  the  answer  is  yes,  then  the  decision  must  be  made  as  to  what  type  of 
research  is  required.  To  answer  this  next  question,  the  analyst  must  appreciate  the 
difference  between  basic  and  applied  research.  These  two  alternatives  represent 
fundamentally  different  approaches  to  the  creation  of  knowledge. 

Basic  and  applied  research  differ  in  their  approach  to  theory.  The  goal  of  basic 
research  is  to  describe  the  world  in  a  truthful  and  accurate  manner.  Developing  and 
advancing  theory  is  thus  the  primary  focus  of  science.  Opposingly,  the  goal  of 
applied  research  is  to  be  useful.  As  such,  theory  is  only  used  to  guide  action,  and  is 
only  important  as  a  tool  to  aid  in  the  effort  to  provide  practical  solutions  to 
problems. 

This  basic  difference  between  the  search  for  truth  and  the  search  for  solutions  leads 
to  differences  in  the  types  of  questions  that  are  asked,  the  standards  of  evidence  to 
which  one  adheres,  and  the  ways  in  which  the  final  results  are  communicated.  The 
analyst  or  researcher  who  seeks  truth,  asks  questions  of  interest  to  him  or  herself, 
and  seeks  to  gather  the  most  precise  data  possible.  This  is  often  done  in  an  attempt 
to  support  or  refute  a  previously  established  hypothesis.  The  results  of  this  exercise 
are  then  reported  at  scientific  meetings  and  in  scientific  journals,  and  rarely  picked 
up  by  the  popular  press.  (These  findings  may  also  be  further  developed  and 
implemented  by  applied  researchers  working  to  solve  a  problem  in  that  area  of 
study.) 

On  the  other  hand,  the  analyst  or  researcher  who  operates  within  the  field  of 
applied  research  will  focus  on  problems  of  current  concern  and  focus  on  those 
variables  that  are  most  easy  to  manipulate  (rather  than  the  most  explanatory).  The 
precision  of  the  data  that  is  used  is  weighed  against  more  practical  considerations, 
such  as  the  cost  involved  in  obtaining  it.  The  applied  researcher  seeks  to  confirm 
that  a  solution  is  useful  in  a  given  situation  or  application.  Results  may  be 
proprietary,  if  a  particular  product  or  process  is  being  studied,  or  may  be  widely 
disseminated. 
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2.01 .08  Is  basic  research  required? 

If  basic  research  is  required,  the  analyst  must  have  an  understanding  of  the  steps 
involved  in  conducting  basic  research.  Basic  research  is  defined  generally  as  an 
original  investigation  for  the  advancement  of  scientific  knowledge,  that  does  not 
have  the  specific  objective  of  applying  this  knowledge  to  practical  problems. 

In  contrast,  applied  research  consists  of  scientific  investigations  intended  to  solve 
practical  problems.  Scientific  knowledge  is  put  into  practice  in  order  to  design  a 
technology  that  will  solve  a  perceived  need  or  problem.  Applied  researchers  are  the 
main  users  of  basic  research.  Thus,  a  new  technology  may  result  from  a  sequence  of 
(1)  basic  research  followed  by  (2)  applied  research  leading  to  (3)  development 


2.02  Opportunity  Costs: 

The  analyst  must  realize  that  the  essence  of  economic  analysis  is  a  systematic 
common  sense  approach  to  die  problem  of  the  efficient  allocation  of  resources.  It  is  a 
formal  decision-making  process  that  is  consistent  with  three  sound  economic 
principles: 

•  All  reasonable  alternative  methods  of  satisfying  a  given  program  objective 
must  be  investigated. 

•  Each  alternative  must  be  considered  in  terms  of  its  full  life  cycle  funding  and 
benefits  implications. 

•  Money  has  value  over  time  as  expressed  by  the  price  it  commands.  In  other 
words,  this  fact  is  included  in  an  analysis  by  expressing  life  cycle  costs  and 
benefits  in  terms  of  their  present  values. 

The  economic  analysis  process  is  a  systematic,  interactive  procedure  for  evaluating 
and  ranking  alternatives  which  meet  an  objective.  The  proper  performance  of  the 
process  requires  the  completion  of  each  of  the  following  six  steps: 

•  Define  the  objective 

•  Generate  alternatives 

•  Formulate  assumptions 

•  Determine  costs  and  benefits 

•  Compare  costs  and  benefits  and  rank  alternatives 
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•  Perform  sensitivity  analysis 

2.02.01  Are  there  other  possibly  relevant  technologies  that  are  not  being 
considered  or  developed? 

Even  though  an  exhaustive  investigation  of  all  possible  avenues  of  research  should 
by  now  have  been  conducted,  it  is  useful  to  pause  for  a  moment  and  consider 
whether  there  may  be  other  possibly  relevant  technologies  that  are  not  being 
developed.  To  accomplish  this,  one  might  use  the  following  protocol.  Once  again, 
begin  by  formulating  an  unbiased  statement  of  the  objective.  The  next  step  is  to 
determine  all  feasible  alternative  methods  of  meeting  that  objective  —  since  the 
ultimate  purpose  of  the  analytical  process  is  to  assist  the  decision  maker  in  making 
the  correct  decision,  it  is  essential  that  all  realistic  alternative  be  considered.  Making 
a  good  decision  is  extremely  difficult  (if  not  impossible)  without  a  full 
understanding  of  all  relevant  options. 

Occasionally  there  will  exist  certain  presumptions  concerning  the  desirability  of  one 
or  more  of  the  options.  There  may  be,  perhaps,  some  administrative  constraints 
(such  as  an  upper  limit  on  spending)  which  tend  to  exclude  certain  alternatives. 
Such  conditions  should  in  no  way  decrease  the  necessity  for  the  analyst  to  do  a 
complete  job.  All  reasonable  and  viable  alternatives  must  be  considered;  otherwise, 
the  value  of  the  analysis  is  seriously  undermined. 


2.02.n2  Is  there  a  cost  associated  with  delaying  this  research? 

The  analyst  must  investigate  each  alternative  to  determine  all  the  costs  occurring 
during  the  entire  project  life  cycle.  Therefore,  s/he  must  explore  and  address  any 
costs  associated  with  delaying  this  research,  and  make  the  appropriate 
recommendations. 


2.02.03  Can  iho  benefits  expected  from  this  prelect  he  Quantified? 

The  essential  aspects  of  an  economic  analysis  are  the  identification  of  all  the 
relevant  inputs  and  outputs  and  the  quantification  of  these  costs  and  benefits  to 
facilitate  evaluation.  Any  economic  analysis  will  involve  considerations  of  both  the 
costs  and  returns  expected  for  each  alternative.  In  general,  there  are  four  types  of 
benefits: 


Direct  Cost  Savings:  Occurs  when  there  is  a  reduction  of  direct  expenditures; 
i.e.,  the  budget  can  be  reduced. 
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Efficiency  /Productivity  Increases:  Occurs  when  there  is  an  increase  in 
productivity  that  can  be  measured  in  dollars  but  does  not  result  in  a  reduction 
of  the  budget 

Other  Quantifiable  Outputs:  Occurs  when  there  is  an  increase  in  productivity 
or  output  that  cannot  be  measured  in  dollars. 

NonOuandfiable  Outputs:  These  are  the  intangibles  that  are  measured  in 
nan-economic  terms  like  goodness,  safety,  or  morale. 


As  mentioned  in  the  above  item,  sometimes  benefits  cannot  be  quantified  in  dollar 
values.  Certain  projects  may  provide  non-quantifiable  benefits  such  as  increased 
retention  rates,  improved  morale,  better  troop  habitability,  and  other  similar 
qualitative  advantages.  Although  these  are  difficult  to  measure,  these  benefits  can  be 
documented  and  assessed  in  the  economic  analysis  through  die  use  of  written 
qualitative  benefit  descriptions  (for  example,  as  portrayed  in  an  opinion  or  attitude 
survey).  This  is  die  least  preferred  method  of  analyzing  benefits  due  to  its  inherent 
lack  of  precision.  However,  under  certain  conditions  this  method  must  suffice,  and 
if  the  following  guidelines  are  observed,  qualitative  statements  can  make  a  positive 
contribution: 

•  Identify  all  benefits  attendant  to  each  alternative  under  consideration.  Give 
complete  details. 

•  Identify  benefits  common  in  kind  but  not  in  extent  or  degTee  among 
alternatives.  Explain  differences  in  detail 

•  Avoid  platitudes,  such  as,  "all  prospective  projects  are  worthwhile  in  that 
they  support  national  defense". 


2.02.03.01  Can  fee  benefits  bo  expressed  Id  forms  of  reduced  life-cycle 
costs? 

One  of  the  benefits  that  an  analyst  must  address  is  reduced  life-cycle  costs.  Generally, 
benefits  reflect  an  expected  return  on  investment  (outputs,  products  or  yields).  The 
benefits  of  each  alternative  should  be  expressed  in  such  a  way  that  the  decision 
maker  is  able  to  compare  various  alternatives. 

This  can  be  done  by  calculating  the  benefit/cost  ratio  (BCR).  The  BCR  is  defined  as: 
benefits  divided  by  costs,  for  each  alternative: 
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BCR  -  Benefits  *■  Costs 


The  BCR  can  be  used  to  express  the  reduced  life-cycle  costs  of  any  alternative. 


2.02.03.02  Can  toe  benefits  bo  expressed  in  terms  of  reduced 
manhours/manpower? 

In  order  for  the  analyst  to  determine  if  the  benefits  to  be  gained  from  his  new 
technology  can  be  expressed  in  terms  of  reduced  manhours/manpower,  he  must 
first  gain  an  understanding  of  how  manhours/  manpower  for  the  civilian  and 
military  work  force  are  computed.  The  next  step  is  to  compare  the  outcome  of  the 
new  technology  manpower  requirements  to  a  standard  baseline  as  determined  by 
the  government. 


2.02.03.03  Can  the  benefits  be  expressed  In  terms  of  Increased  productivity? 

Projects  for  modernization,  rehabilitation,  consolidation,  and  other  related  goals  can 
often  generate  an  increase  in  the  efficiency  of  operations  or  productivity.  Such 
increases  are  extremely  beneficial  and  should  be  included  in  a  benefit/cost  analysis 
when  they  exist  However,  it  is  important  to  note  that  benefits  of  this  type  are 
frequently  confused  with  direct  cost  savings  because  they  are  easily  quantified  in 
dollar  terms.  But  in  reality,  they  are  not  equivalent,  and  the  analyst  should 
understand  the  fundamental  difference  dearly. 

An  increase  in  effidency  or  productivity  implies  only  one  filing  -  the  ability  to  do 
more  work  within  the  "xisting  manpower/ funding  level.  The  only  way  to  translate 
an  effidency  /productivity  increase  into  direct  cost  savings  is  to  effect  a  reduction  in 
force  (RIF)  which  lowers  the  required  funding  level  However,  a  REF  is  not  usually 
intended  as  the  mandated  result  of  a  government  project,  and  thus  some  other 
means  of  quantifying  efficiency/productivity  benefits  must  be  used. 

The  solution  to  the  problem  is  really  a  simple  matter  of  semantics.  An 
effidency/productivity  increase  which  translates  into  a  labor  time  savings  of  two 
man-years  is  a  benefit.  Its  value  may  be  defined  as:  what  it  would  cost  the 
government  to  buy  an  additional  two  man  years  of  labor.  This  cost  should  also 
reflect  the  appropriate  rate  for  leave  and  fringes,  because  the  value  of  benefits 
should  reflect  the  actual  total  cost  to  the  government  of  providing  two  man-years  of 
work. 

One  very  important  caveat  must  be  mentioned  at  this  point  In  order  to  daim  an 
effide ncy/ productivity  increase  as  a  valid  benefit  there  must  be  a  documented  Hfifid 
for  the  increased  workload  capacity.  In  other  words,  there  must  be  an  alternative  use 


to  which  the  "new"  manpower  resources  can  be  put,  such  as  reducing  a  backlog  of 
maintenance.  Lacking  this,  there  is  no  benefit  -  or  at  least  no  quantifiable  benefit  - 
derived  from  the  project  Documentation  of  this  fact  must  be  complete  and  explicit 
in  the  benefit/cost  analysis. 


2.02.03.04  Coo  tbo  benefits  bo  expressed  to  terms  of  reduced  training  costs? 

In  an  environment  of  downsizing  and  severely  constrained  budgets,  users  will  be 
very  receptive  to  technology  which  can  demonstrate  measurable  training  cost 
benefits.  Therefore,  it  is  important  that  considerable  effort  be  expended  in 
demonstrating  this  important  benefit  Before  answering  this  question,  the  analyst 
needs  to  establish  his/her  baseline,  that  is,  what  are  the  current  costs  of  training 
using  the  prooesses/equipment/methodologies  to  be  replaced?  If  these  costs  are  not 
readily  available,  HQ  ATC  should  be  consulted  for  assistance.  Considerable  cost  data 
reside  there.  It  is  important  to  understand  what  specific  costs  are  included  in  these 
data  in  order  to  ensure  that  equivalent  cost  categories  are  used  for  comparison 
purposes.  Cost  categories  might  indude  such  things  as  personnel  (trainer,  trainee, 
staff  and/or  other  administrative  overhead),  facilities,  equipment,  supplies,  books, 
travel,  per  diem,  etc  In  estimating  the  training  costs  of  the  training  alternatives 
being  pursued,  manufactureis/suppliexs  should  be  good  sources  of  information,  so 
the  analyst  should  be  sensitive  to  the  availability  of  such  information  during 
conduct  of  the  market  survey.  Also,  seek  out  activities  using  the  same  oi 
comparable  technology. 


2.02.03.05  Cao  tbo  benefits  bo  expressed  In  forms  of  reduced  training  time? 

This  is  significant  to  the  user  because,  assuming  the  training  is  at  least  as  effective  as 
that  currently  being  provided,  reduced  training  time  can  be  directly  translated  into 
increased  opportunity  to  be  productive  on  the  job.  In  computing  this  benefit,  the 
same  process/procedures  as  discussed  above  in  2.02.03.04  should  be  used,  and,  in 
fact,  reduced  training  time  should  contribute  to  reduced  training  costs. 


2.02.03.06  Can  die  benefits  be  expressed  in  terms  of  Increased  student 
achlevement/performance? 

Unless  a  source  can  be  found,  such  as  an  activity  experienced  in  use  of  comparable 
technology,  this  question  may  be  difficult  to  answer  with  precision  until  some 
testing  has  been  performed.  Preceding  this,  however,  a  baseline  can  be  established, 
again  using  HQ  ATC  as  a  logical  source  if  data  are  not  otherwise  readily  available. 
And  of  course  the  real  answer  will  come  when  the  user  is  requested  to  provide 
feedback  after  a  set  period  of  deployment  and  operation  of  the  technology. 


2.02.04  Have  life-cycle  cost  estimates  been  prepared  for  the  technology 
under  consideration? 

The  analyst  must  presume  that  for  all  requirements  there  are  alternative  choices  or 
trade-offs  for  reaching  the  objective,  even  if  one  of  the  options  is  to  maintain  a 
status  quo,  or  to  do  nothing.  Thus,  the  economic  analysis  that  must  be  done  should 
incorporate  the  full  life  cycle  implications  of  each  alternative,  in  terms  of  costs  and 
benefits.  This  is  accomplished  by: 

•  Focusing  thinking  —  both  formal  and  informal 

•  Surfacing  assumptions  —  both  hidden  and  presumed  —  and  identifying  their 
logical  implications 

•  Analyzing  all  costs,  benefits  and  their  timing 

•  Providing  an  effective  communications  vehicle  for  all  considerations  in 
support  of  the  decision. 


Life-cycle  cost  estimates  must  include  all  costs  in  the  appropriations  accounts  of 
Research,  Development,  Test,  and  Evaluation  (RDT&E);  Procurement;  Military 
Construction  (MILCON)  (if  applicable);  Operations  and  Maintenance  (O&M);  and 
Military  Personnel  for  the  expected  life  of  the  system. 


2.02.05  Has  the  expected  amortization  period  for  the  costs  associated 
with  the  research  been  determined? 

Amortization  is  defined  as  the  gradual  reduction  of  the  balance  in  an  account 
according  to  a  specified  schedule  of  time  and  amounts.  It  is  usually  the  provision 
for  extinguishing  a  debt,  including  interest,  by  means  of  a  sinking  fund  or  other 
form  of  payment 


2.02.06  Has  expected  retum-on-lnvestment  been  calculated? 

The  first  step  that  the  analyst  must  take  in  comparing  a  proposed  alternative  against 
the  status  quo  is  to  calculate  the  savings  to  investment  ratio  (SIR).  It  is  the  amount 
of  savings  generated  by  each  dollar  of  investment  Since  all  government  economic 
analyses  must  take  the  time  value  of  money  into  account,  the  SIR  is  mathematically 
defined  as: 
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all  feasible  alternatives  or  ways  of  meeting  the  objective.  In  other  words,  all  the 
alternative  means  by  which  the  objective  can  be  reached  must  be  identified.  It  is 
important  for  the  analyst  to  note  that  the  status  quo,  or  the  cost  erf  doing  nothing,  is 
often  a  viable  alternative  and  must  be  assessed. 


2J3.02  If  alternatives  are  feasible,  have  their  costs  been  calculate*? 


See  2.02.04  above  and  calculate  life-cycle  costs  for  each  alternative  being  considered. 


2.03.03  If  alternatives  are  feasible,  has  their  effectiveness  been 
determined  or  estimated? 

Although  there  may  be  other  ways  of  measuring  effectiveness  of  the  technology,  it  is 
essential  that  the  levels  of  proficiency  anticipated  to  be  attained  by  the  trainees  be 
assessed/estimated  far  each  alternative.  As  indicated  in  2.02.03.06  above,  precision 
cannot  be  expected  at  this  point;  however,  like-type  estimates  are  acceptable  for  each 
of  the  alternatives  being  considered.  If  any  of  the  alternatives  do  not  appear  to  offer 
increased  levels  of  proficiency,  they  should  probably  be  dropped  from  further 
consideration. 


2.03.03.01  For  each  alternative,  have  levels  of  proficiency  anticipated  to  be 
attained  by  ihe  trainees  been  assessed/estimated? 

The  analyst  must  realize  that  the  dearest  finding  across  a  range  of  studies  on 
technology  implementation  is  that  the  selection  and  retention  of  well-qualified 
personnel  is  the  most  critical  human  resource  issues  to  address. 

Therefore,  for  each  alternative,  it  is  important  for  the  analyst  to  assess  the  levels  of 
profidency  antidpated  to  be  attained  by  the  trainees.  While  to  some  extent  it  is 
obvious  that  some  training  needs  to  be  provided,  it  is  less  obvious  who  should  be 
trained  (supervisors  or  operators  of  the  technology?),  how  many  people  should  be 
trained  (should  training  be  provided  to  everyone  or  just  to  a  few  lead  individuals?), 
and  what  methods  should  be  used  (dassroom  courses  or  on  the  job?). 
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2.0* 


Risk  Assessment 


Performing  a  detailed  risk  analysis  is  useful  to  the  analyst  for  the  following  four 

reasons: 

•  Resources  (time,  money,  and  management  attention)  can  be  allocated  based 
on  each  project's  risk. 

•  Candidate  tasks  for  prototyping  can  be  identified.  Prototyping  risky  software 
functions  or  modules  is  often  the  most  powerful  method  available  to  the 
analyst  to  reduce  risk. 

•  A  risk-optimized  schedule  can  be  generated.  A  risk-optimized  schedule 
attempts  to  schedule  risky  tasks  as  early  as  possible  in  the  project  to  allow 
sufficient  time  for  recovery  in  the  event  of  failure. 

•  Contingency  planning  is  possible. 


2.04.01  Is  there  tbs  likelihood  that  the  tsctmolooy  will  not  work  as 
intended? 


To  varying  degrees,  the  answer  to  this  question  is  probably  always  "yes."  The 
challenge  to  the  analyst  is  to  judge  to  what  degree  such  is  the  case,  and  then 
determine  the  users'  willingness  to  accept  that  degree  of  risk.  For  example,  if  the 
odds  are  judged  to  be  50/50,  the  user  may  find  this  too  risky  to  continue  pursuit  of 
the  technology.  However,  if  the  odds  are  estimated  to  be  30/70  yes/no,  the  user  may 
be  willing  to  accept  that  degree  of  risk. 

Risk  assessment  is  an  ongoing  process  that  should  take  place  throughout  the 
development  process.  During  the  planning  phase,  toe  risk  assessment  process 
should  focus  on  issues  tuat  relate  to  toe  feasibility  of  the  project  Risk,  generally 
speaking,  is  directly  related  to  the  development  cost  and  schedule.  If  cost  is  a  critical 
issue,  toe  risks  impacting  resources  should  be  explored,  and  those  alternatives  that 
can  eliminate  or  reduce  these  risks  to  an  acceptable  level  should  be  identified.  If 
scheduling  is  a  critical  issue,  risks  impacting  toe  schedule  have  to  be  identified,  and 
eliminated  or  reduced  before  the  project  begins. 


2.04.02  Will  tfia  technology  provide  safety  risks  to  the  user? 

In  determining  the  need  for  the  new  technology,  the  analyst  must  determine 
whether  it  significantly  contributes  to  system  risk.  He  must  answer  the  question  of 
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whether  or  not  the  new  technology  could  cause  loss  of  life,  personnel  injury, 
mission  failure,  or  equipment  failure. 


2.04.03  Will  the  technology  be  of  uncertain  reliability  to  the  user? 

The  analyst  must  be  aware  that  there  are  four  forms  of  warranties  that  can  be 

invoked  contractually: 

•  Commercial  warranty  —  fixed-price  warranty  where  the  seller's  liability  is 
limited  to  replacing  faulty  parts  with  a  like  item  in  working  order.  The  seller 
does  not  assume  any  other  form  of  liability  whatsoever.  This  type  of 
warranty  is  straightforward  and  has  been  used  for  years  in  software  contracts. 
Literally,  it  is  a  "let  the  buyer  beware"  warranty  where  die  seller  is  liable  only 
for  replacement  parts. 

•  Extended  warranty  —  fixed-price,  extended  duration  commercial  warranty 
where  the  seller  fixes  all  legitimate  failures  at  no  additional  cost  to  the  buyer 
throughout  the  life  of  the  contract  It  is  generally  applied  as  a  maintenance 
contract  The  seller  repairs  or  replaces  die  failed  version  with  a  new  release 
that  fixes  the  problem.  The  buyer  waives  all  right  to  hold  the  seller  liable  for 
damages  outside  the  seller's  control.  Timeliness  of  repair  and  responsiveness 
of  the  seller  are  sometimes  inadequate. 

•  Reliability  warranty  —  fixed-price,  extended  duration  specification  warranty 
where  the  seller  fixes  only  legitimate  failures  in  excess  of  an  allowed  number 
based  on  specified  reliability  at  no  additional  cost  to  the  buyer  throughout  the 
life  of  the  contract 

•  Performance  guaranty  -  fixed-price,  extended  duration  guarantees  where  the 
seller  agrees  to  meet  specific  performance  requirements.  In  case  of  failure,  the 
seller  is  held  liable  for  free  maintenance  and/or  redesign  and  retrofit  at  no 
cost  to  the  buyer.  Performance  measures  could  indude  such  items  as  mean 
time  between  failure,  response  time,  speed,  accuracy,  range,  and  resolution. 


2.04.04  Is  the  technological  risk  high? 

In  some  projects,  the  amount  of  risk  is  acceptable  when  compared  to  the  antidpated 
benefits  of  the  effort.  A  project  may  undertake  to  develop  or  explore  the  possibility 
of  achieving  a  new  capability  (such  as  in  artifidal  intelligence  applications).  Or,  as 
another  example,  the  development  may  be  a  prototype  effort  to  define  or  eliminate 
a  high-risk  area  critical  to  a  subsequent  development  effort  A  prototype  effort  is 
undertaken  to  reduce  this  technical  risk  before  major  resources  are  committed  to  the 
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development  of  the  actual  system. 


2.04.05  Is  then  a  risk  that  the  user  will  not  accept  the  product? 

Controlling  user  expectations  is  an  important  part  of  the  development  process.  No 
potential  user,  or  customer,  has  a  dear,  complete,  and  firm  concept  of  exactly  what 
the  final  product  will  be  like  at  the  end  of  the  project  Their  expectations  for  the 
final  product  develop  and  mature  during  the  project  based  primarily  on  discussions 
with  the  analyst  and  his  team.  If  the  analyst  emphasizes  the  "bells  and  whistles"  of 
die  design  during  development,  the  user,  or  customer,  will  expect  the  "bells  and 
whistles"  and  will  be  disappointed  if  they  are  not  delivered.  On  the  other  hand,  if 
the  analyst  emphasizes  the  functional  aspects  of  the  program,  die  user,  or  customer, 
will  be  pleasantly  surprised  when  he  is  delivered  what  was  promised  along  with 
some  beneficial  additional  features.  In  short,  keep  die  user,  or  customer,  "focused 
on  die  steak,  not  the  sizzle!" 


L04J5.01  At  least  annually,  have  yon  asked  the  nser  to  verify  and 
re-valldate  commitment  to  the  prelect? 

The  analyst  must  establish  a  post-implementation  review  plan  that  at  least 

addresses  die  following: 

•  Establish  a  time-frame  for  the  first  post-implementation  review.  Provide 
figures  for  die  estimated  system  life,  and  a  schedule  for  the  remaining 
post-implementation  reviews. 

•  A  post-implementation  review  should  be  conducted  6  to  12  months 
following  system  installation  to  ensure  that  the  system  is  functioning  as 
designed. 

•  Subsequent  reviews  should  be  conducted  at  50  per  cent  and  80  per  cent  of  the 
system's  life,  or  at  least  every  three  years,  whichever  is  less. 


2.05  Software  Supportablllty: 

The  analyst  must  consider  the  following  dominant  characteristics  of  software 
supportability: 

•  The  requirements  are  diverse.  In  other  words,  the  system  will  perform  a 
variety  of  functions  that  cannot  be  described  in  just  a  few  pages. 
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•  The  software  is  not  an  off-the-shelf  commercial  product,  although  there  may 
be  off-the-shelf  commercial  products  in  the  custom  system. 

•  The  users  of  the  system  are  a  diverse  group  performing  many  different  tasks. 

•  The  users,  or  customers,  are  different  from  those  responsible  for  acquiring  the 
system.  For  example,  one  would  not  expect  a  teller  to  be  involved  in 
acquiring  the  system  for  a  bank.  A  teller  might,  however,  be  involved  in 
testing  the  system. 

•  The  developer  is  different  from  the  acquired.  Normally,  the  acquired  has  a 
contractor  develop  the  system. 


2.05.01  Will  software  be  needed  to  support  the  technology? 

The  analyst  must  be  aware  that  project  requirements  are  generally  of  two  types:  (1) 
the  statement  of  need  from  the  customer  or  the  end-user  of  the  system,  and  (2)  the 
requirements  necessary  to  manage  the  project  Irrespective  of  management 
requirements,  adequate  attention  to  the  statement  of  need,  or  performance 
requirements,  is  an  essential  prerequisite  for  an  acceptable  project  The  requirements 
statement  is  the  foundation  for  a  software  development  decision. 


2.05.02  Are  the  programming  language  and  software  compatible  with 
existing  systems/hardware  used  by  both  the  developing  agency 
and  the  user(s)? 

In  order  to  properly  answer  this  question,  the  analyst  must  be  aware  that  at  the 
system  and  segment  level  he  is  dealing  with  entities  which  are  normally  a 
combination  of  software  and  hardware. 

A  system  is  a  collection  of  hardware,  software,  material,  facilities,  personnel,  data, 
and  services  needed  to  perform  a  designated  function  with  specified  results.  A 
segment  is  a  grouping  of  elements  that  are  closely  related  and  often  physically 
interfaced. 

Each  segment  consists  of  some  combination  of  hardware  configuration  items 
(HWCIs)  and  computer  software  configuration  items  (CSCIs).  Each  configuration 
item  is  normally  responsible  for  a  single  top-level  function  and  is  normally 
developed  by  one  prime  contractor. 

Hardware  configuration  items  consist  of  hardware  components  and  software 
components.  Hardware  components  can  be  further  divided  into  sub-assemblies  and 
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Software  configuration  items  consist  of  computer  software  components  (CSCs)  and 
computer  software  units  (CSUs).  As  much  as  possible,  configuration  items  should  be 
selected  in  a  fashion  which  will  allow  parallel  development  and  testing  (i.e.,  as  few 
dependides  between  configuration  items  as  possible). 


2.05.03  Is  the  software  mature  and  proven? 

In  order  for  the  software  to  be  judged  mature  and  proven,  the  analyst  mtist  check  for 
software  quality  assurance.  Software  defects  can  be  divided  into  four  broad 
categories: 

•  Requirements  defects 

•  Design  defects 

•  Code  defects 

•  User  documentation  defects 

The  cost  to  find  and  repair  software  defects  is  usually  highest  for  requirements 
defects,  and  usually  lowest  for  documentation  defects.  Requirements  defects 
normally  result  in  improper  design,  code,  and  documentation.  Design  defects  result 
in  improper  code  and  documentation.  Code  defects  can  be  limited  to  code  problems 
that  have  resulted  through  human  error. 


2.05.04  Is  ths  software  maintainable  for  Ore  Ilfs  of  tho  prelect? 

The  analyst  must  be  aware  that  the  government  nornudly  requires  the  contractor  to 
maintain  the  system  for  a  period  of  time  after  it  is  delivered  and  in  operation.  This 
is  usually  a  period  of  one  to  two  years.  If  transition  planning  is  not  accomplished, 
the  government  will  have  problems  downstream.  The  key  transition  issues  are: 


Will  the  contractor  be  required  to  provide  maintenance  and  support  and  for 
what  period  of  time? 

Will  the  government  want  to  transfer  maintenance  and  support  of  the 
system  to  a  new  supplier,  either  at  the  successful  conclusion  of  the 
development  or  at  a  later  time? 
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Will  the  system  be  recompeted  at  a  later  time  to  provide  for  major 
enhancements  or  a  new  maintenance  supplier? 


•  Will  the  government  require  that  system  maintenance  and  support  be 
transferred  to  the  user  organization? 

Appropriate  documentation  is  also  necessary  to  support  any  transition,  and  the 
maintenance  support  facility  must  receive  adequate  consideration. 


2.05.04.01  Have  maintenance  requirements  been  Identified? 

Maintenance  is  a  support  function  that  occurs  routinely  in  a  software  project. 
Maintenance  involves  correcting  faults,  modifying  the  system  to  incorporate  new 
requirements,  incorporating  changes  that  add  capability,  and  modifications  to 
enhance  performance. 

Maintenance  requirements  must  be  planned  at  the  beginning  of  the  project. 
Decisions  about  documentation  and  delivered  software  (especially  support  software 
and  test  cases)  are  very  important  if  responsibility  for  maintenance  or  upgrading  the 
system  is  to  be  transferred  to  the  government.  In  many  cases,  the  government 
representatives  may  not  realize  that  these  preparations  must  be  made,  or  may 
assume  that  support  of  the  system  will  be  provided  by  die  contractor.  When  this 
occurs,  the  government  may  be  left  with  a  system  that  is  improperly  documented 
and  difficult  to  maintain. 


2.05.04.02  Has  a  determination  been  made  as  to  who  will  be  responsible  for 
maintaining  the  technical  currency  of  the  information  presented 
by  the  software? 

The  determination  of  who  will  be  responsible  for  the  management  of  the  technical 
information  or  content  of  the  software  is  an  important  consideration.  This  task 
involves  the  gathering  and  retention  of  all  relevant  data.  The  management  of  this 
database  of  information  requires  standardized  data  definitions,  data  management 
facilities,  and  a  staff  in  some  cases,  to  ensure  that  the  data  is  promptly  obtained, 
properly  checked,  accurately  entered  into  the  data  base,  and  effectively  managed.  If 
managed  properly,  the  data  base  will  always  be  technically  current 
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2.05.04.03  For  soflwara  developed  by  a  contractor,  will  the  software  be 
owned  by  the  user  without  proprietary  restrictions? 

Since  a  considerable  quantity  of  software  is  either  developed  under  federal 
government  contract,  or  licensed  to  the  federal  government,  the  government 
procurement  regulations  are  relevant  to  any  discussion  of  software  procurement. 
The  general  policy  provisions  and  standard  contract  clauses  found  in  these 
regulations  govern  all  transactions  by  which  software  and  its  documentation  are 
acquired  by  the  government 

The  analyst  must  ensure  that  a  special  clause  is  included  in  the  contract  that  states 
that  the  government  will  own  unlimited  rights  to  the  technical  data  and  computer 
software  developed  on  the  contract  This  means  that  the  government  as  the  buyer, 
can  use,  duplicated,  or  disclose  this  technical  data  and  computer  software  in  whole 
or  in  part,  in  any  manner  and  for  any  purpose  whatsoever,  and  can  grant 
permission  for  others  to  do  the  same.  "Technical  data",  in  this  context,  refers  to  any 
recorded  information  —  regardless  of  form  or  character  —  of  a  scientific  or  technical 
nature.  It  does  not  indude  cost,  pricing,  and  management  data,  or  other  information 
incidental  to  the  contract  administration.  "Computer  software”  refers  to  the  set  of 
instructions  that  cause  a  digital  computer  to  operate,  and  all  documentation 
associated  with  that  process. 

This  clause  requires  die  contractor  to  identify  any  restricted  rights  on  technical  data 
and  computer  software  that  will  be  used  in  the  generation,  production,  or  operation 
of  items  to  which  die  contractor  has  unlimited  rights.  For  example,  if  a  proprietary 
software  tool  is  needed  to  generate  the  operational  software,  it  must  be  identified  so 
that  a  price  and  appropriate  rights  for  the  software  can  be  negotiated  prior  to 
contract. 

Occasionally,  this  dause  is  coupled  with  a  deferred  delivery  dause  that  allows  the 
government  to  purchase  the  rights  to  the  technical  data  and  computer  software  and 
take  delivery  for  a  fixed  price  at  some  later  time  during  the  acquisition  process. 
Such  clauses,  though  rare,  can  substantially  improve  the  government's  leverage, 
enabling  purchase  of  items  at  a  later  time  when  additional  funds  may  become 
available. 


2.05.04.04  Have  the  costs  of  maintenance  been  estimated? 

The  costs  of  software  maintenance  should  be  estimated  for  both  in-house  and 
contractor  maintenance,  and  the  costs  compared.  The  system's  size,  complexity,  and 
lines  of  code  will  be  some  of  the  factors  driving  these  costs.  The  analyst  should 
research  maintenance  costs  associated  with  systems  with  similar  characteristics. 
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2.05.04.05  Hava  maintenance  manhours  been  estimated? 


Software  maintenance  manhours  will  make  up  the  large  majority  of  the  costs 
estimated  for  in-house  maintenance  in  2.05.04.04  above.  As  with  maintenance  costs, 
comparable  systems  should  be  analyzed  to  assist  in  estimating  manhours  required. 


2.05.04.06  Are  maintenance  manhours  supportable  with  existing  manpower? 

This  question  applies,  of  course,  only  if  the  decision  has  been  made  to  pursue  the 
option  of  maintaining  the  software  with  in-house  resources.  If  the  required 
manhours  are  not  supportable,  the  option  to  maintain  the  software  through 
contractor  resources  should  be  pursued  or  reconsidered 

2.06  Hardware  Supportabillty: 


2.06.01  Hava  needs  lor  supporting  hardware  been  identified? 

The  analyst  must  be  able  to  identify  the  needs  of  the  supporting  hardware.  One 
technique  is  the  use  of  "capacity  management”.  This  allows  the  analyst  to  ensure 
that  the  new  system  is  properly  designed  and  configured  to  give  efficient 
performance,  and  that  it  has  sufficient  resources  to  support  operating  work  loads. 
As  part  of  this  process,  future  work  loads  and  required  user  service  levels  (e.g., 
system  availability)  are  forecasted.  Furthermore,  system  configurations  necessary  to 
meet  the  demands  are  proposed,  modeled,  and  tested. 

Stress  testing  is  an  integral  part  of  capacity  management  Its  purpose  is  to  ensure  that 
the  total  system  will  successfully  process  workloads  expected  during  peak 
production  periods  and  other  extreme  conditions  and  to  determine  the  point  at 
which  major  system  resources  (e.g.,  processor,  channels,  primary  storage,  etc)  will  be 
exhausted. 


2.06.02  Will  user's  existing  hardware  be  used? 

The  analyst  must  identify  the  existing  hardware  to  still  be  used  and  any  changes 
required  to  exercise  the  new  technology. 


2.06.03  is  the  hardware  compatible  with  existing  systems? 

The  analyst  must  identify  the  hardware  that  is  compatible  with  the  existing  systems 
and  specify  any  required  changes. 


2.06.04  is  the  hardware  mature  and  proven? 

The  analyst  must  ensure  that  the  hardware  is  mature  and  proven.  This  can  be 
accomplished  by  requiring  that  all  components  of  the  system  be  identified  and 
documented,  and  that  changes  to  these  components  be  controlled,  recorded,  and 
reported.  Among  the  many  components  subject  to  these  requirements  are  system 
test  plans  and  the  associated  test  results,  and  system  manuals  (e.g.,  user  manuals, 
operator  manuals,  and  maintenance  manuals).  These  requirements  are  designed  to 
ensure  that  the  system  meets  user  requirements  and  can  be  operated  and 
maintained  efficiently  and  effectively. 

Another  area  for  the  analyst  to  check  for  hardware  maturity  is  the  results  of  the  test 
plan.  Test  procedures  should  specify  exactly  what  test  inputs  to  provide,  what  steps 
to  follow,  what  outputs  to  expect,  and  what  criteria  to  use  in  evaluating  the  outputs. 
If  any  of  these  elements  are  absent,  the  test  procedures  are  to  be  considered 
inadequate. 


2.06.05  Is  the  hardware  maintainable  tor  the  life  of  tea  project? 

Reliability  describes  the  system  in  terms  of  its  probable  failure  rate.  Useful  measures 
may  be:  the  mean  time  between  failures,  the  number  of  service  calls  per  year,  or  the 
percentage  of  refusals  per  warehouse  requests. 


2.06.05.01  Is  logistics  support  capability  In  place? 

To  ensure  that  a  logistics  support  capability  is  in  place,  the  analyst  must  be  cognizant 

of  the  following: 

•  Maintenance-  The  analyst  must  be  aware  of  the  status  of  maintenance 
arrangements  for  the  new  system’s  equipment  and  software. 

•  Personnel*  The  analyst  must  be  aware  of  the  status  of  personnel  required  to 
operate,  maintain,  and  use  the  system. 

•  Facilities  -  The  analyst  must  be  aware  of  the  status  of  facility  preparation  for 
this  project 
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2.07  User  Training: 

The  analyst  must  be  aware  of  all  types  of  maintenance  and  the  operator  training 
hardware,  devices,  visual/audio  training  aids  and  related  software  which  satisfy  the 
following: 

•  Used  to  train  maintenance  and  operator  personnel  by  depicting,  simulating  or 
portraying  the  operational  or  maintenance  characteristics  of  an  item,  system 
or  facility. 

•  Training  equipment  must,  by  their  nature,  be  kept  consistent  in  design, 
construction  and  configuration  with  such  items  in  order  to  provide  required 
training  capability. 


2.07.01  Will  users  or  trainers  need  special  training  to  use  the  technology? 

The  analyst  must  consider  an  overall  training  strategy  that  would  include  the  need 
for  system  training  devices  and  embedded  training  requirements.  S/he  must 
consider  new  ec  lipment  training,  operator  and  maintenance  training,  technical 
manuals  and  training  materiel  requirements  for  both  institutional  and  unit 
training. 

2.07.02  Has  user  training  been  specified  as  a  deliverable  for  this  prelect? 

The  analyst  must  be  aware  of  user  training  issues  regarding  the  new  technology.  If  it 
is  a  contract  deliverable,  the  analyst  must  realize  that  user  training  stresses  the  role 
and  responsibilities  of  the  user  in  the  operation  of  the  system.  The  overall 
responsibility  is  to  ensure  that  the  vendor  delivers  the  user  training  as  contracted. 

User  training  can  consist  of  describing  the  overall  objectives  and  benefits  of  the 
system  and  explaining  user  procedures  in  detail.  These  procedures  include; 

•  Data  collection  and  preparation, 

•  Data  transaction  and  submission  or  data  transmission, 

•  Data  base  maintenance  -  frequency,  additions,  deletions,  changes,  error 
corrections, 

•  Reviewing  and  using  system  output  and/or  querying  the  data  base. 
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•  Balancing  and  control  of  output, 

•  Security /internal  controls, 

•  User  documentation, 

•  Job  setup  instructions, 

•  Operating  instructions  including  restart  and  recovery  procedures, 

•  Quality  control, 

•  Output  preparation  including  bunting,  decollating,  and  distribution, 

•  User  liaison. 


2J7J3  Has  an  OPR  bran  designated  to  deeign/delher  the  user  training? 

It  is  important  for  the  analyst  to  be  cognizant  of  who  has  been  designated  as  the 
Office  of  Primary  Responsibility  (OPR)  for  the  design  and  delivery  of  die  user 
training.  The  analyst  must  be  prepared  to  indude  him  or  her  as  s  major  participant 
in  the  new  technology  development  plan.  It  it  important  that  training  be  addressed 
adequately  and  that  it  occur  before  the  system  is  certified  as  acceptable. 


2.08  Flexibility: 

Configuration  management  is  a  very  important  topic  on  government  projects  in 
order  to  ensure  the  flexibility  of  the  final  product  Government  projects  are  typically 
quite  large  and  complex  They  often  involve  concurrent  development  by  multiple 
vendors.  The  government  must  insist  that  configuration  control  procedures  be 
carefully  followed  throughout  the  project 


2.08.01  Can  tbs  technology  Incorporate  Job-related  technological  updates 
(exchanges  to  equloment  and/or  procedures)? 

Job-related  changes  and  updates  are  not  discouraged,  but  must  be  planned  for  and 
controlled.  Classes  of  changes  as  identified  by  foe  government  are: 

•  Class  I:  These  changes  have  a  significant  impact  on  the  software's  user 
interface,  technical  performance,  efficiency,  oost,  or  schedule.  In  addition,  any 


changes  involving  modifications  to  the  interfaces  between  configuration 
items  would  normally  be  considered  Class  I  changes. 


*  Class  H:  These  changes  are  considered  minor.  Examples  include  editorial 
changes  to  documentation,  changes  to  the  internal  structure  of  software  if  the 
impact  is  localized  and  not  apparent  to  the  user,  and  material  substitutions 
for  hardware. 


2.08.01.01  Are  mechanisms  available/ln  place  to  ensure  timely  notification  of 
changes? 

The  analyst  must  be  aware  of  the  mechanisms  available/in  place  to  ensure  timely 

notification  of  changes.  Generally,  five  forms  are  available  for  use  during  the 

change  control  process: 

•  The  Engineering  Change  Proposal  CECP):  This  form  is  used  by  the  contractor 
to  propose  Class  I  changes.  It  includes  a  description  of  the  change, 
justification  for  the  change,  configuration  items  effected  by  the  change,  the 
impact  on  integrated  logistic  support  and  operational  effectiveness,  impact  on 
the  project's  cost  and/or  schedule,  and  results  of  trade-off  analysis  looking  at 

alternate  solutions,  if  appropriate. 

•  The  Advanced  Change/Studv  Notice  (ACSN):  This  form  is  used  to  seek 
government  approval  to  expend  engineering  effort  to  prepare  a  formal 
Engineering  Change  Proposal.  It  includes  the  need  for  the  change,  the 
configuration  item(s)  effected,  alternatives  to  be  considered,  and  a  rough  cost 
estimate.  A  preliminary  ECP  may  be  used  in  place  of  an  ACSN.  In  either 
case,  the  form  is  submitted  to  the  government  which  may  reject  the  change  or 
authorize  the  necessary  engineering  effort  to  prepare  a  formal  ECP. 

•  The  Specification  Change  Notice  (SCN):  This  form  is  used  to  propose, 
transmit,  and  record  changes  to  a  specification.  ECPs  which  will  affect  a 
specification,  e.g..  System /Segment  Specification,  must  have  a  SCN  form 
attached. 

•  A  Request  for  Deviation/Waiver  This  form  is  used  to  request  and  document 
temporary  departures  from  the  specifications  when  a  permanent  departure 
would  not  be  acceptable.  Although  this  form  primarily  applies  to  hardware, 
software  development  projects  may  use  it  if  one  plans  to  make  preliminary  or 
phased  deliveries  where  the  software  as  initially  delivered  will  not  meet  the 
full  requirements. 


453 


2.08.01.02  Have  methods  and  requirements  for  Incorporating  new  data  Into 
the  system  been  established? 

Methods  and  requirements  for  incorporating  new  data  into  the  system  should 

encompass  the  following: 

•  All  processor  instructions/operation  codes  (e.g.,  load,  add,  subtract,  etc.)  will 
produce  known  responses  by  the  computer; 

•  Read,  write  and  execute  access  rights  of  the  user  will  be  verified  each  time  a 
computer  instruction  is  executed; 

•  Unauthorized  attempts  to  change,  circumvent  or  otherwise  violate  system 
security  features  will  be  detectable  and  will  abort  or  suspend  the  operation 
running;  and 

•  An  audit  log  or  file  will  be  maintained  as  a  history  of  system  use. 


2.08.01.03  Has  an  OPR  for  Implementing  the  changes  been  Identified? 

It  is  absolutely  necessary  that  an  Office  of  Primary  Responsibility  (OPR)  for  the 
implementation  process  be  designated  early  in  die  new  technology  development 
process.  This  is  an  important  issue  for  due  analyst  to  address,  because  in  any 
technological  innovation  process,  there  are  bound  to  be  many  decision  makers  who 
play  a  variety  of  roles. 

One  particular  role  that  has  received  special  attention  is  that  of  a  product  or  process 
champion.  This  is  an  advocate  that  who  works  at  gaining  support  for  change.  In 
effect,  champions  provide  a  conduit  for  ideas  from  outside  the  organization,  and 
work  to  continuously  keep  change  and  innovation  on  the  organization's  internal 
agenda.  Besides  acting  as  an  advocate,  the  champion  performs  die  important  service 
of  linking  together  the  different  phases  of  the  technological  innovation  decision 
making  process.  The  complex  process  of  innovation  in  unlikely  to  succeed  without 
an  individual  or  group  of  individuals  to  take  responsibility  for  coordinating  its 
implementation. 


2.08.02  Can  the  research  be  redirected  or  re-handled  at  critical  points.  If 
necessary? 

A  project  management  schedule,  or  a  set  of  milestones,  must  be  developed  very 
early  in  the  new  technology  life-cycle  process.  This  will  allow  toe  analyst  to  redirect 


or  rechannel  the  research  at  critical  points  if  it  becomes  necessary  due  to  any 
unanticipated  changes.  A  management  plan  of  action  should  provide  a  description 
of  the  tasks,  work  effort  and  schedule  required  to  meet  the  goals  of  the  research. 

During  execution  of  the  effort,  technical  performance  measurement  should  be 
accomplished  using  the  parameters  identified  in  the  management  plan  and  the 
subtler  parameters  supporting  it  to  compare  achievement-to-date  against  the 
projected  schedule  and  work  effort  The  technical  performance  measurement  tasks 
and  their  correlation  with  contractual  cost/schedule  elements  shall  be  defined  to 
permit  assessment  of  the  program  effort  in  terms  of  the  schedule  and  cost  of  work 
increments  and  comparison  of  planned  value  of  work  accomplished  with  both  the 
planned  value  of  work  scheduled  and  the  actual  cost  of  work  accomplished.  This 
correlation  is  critical  to  risk  identification  and  assessment. 

Knowledge  of  this  process  allows  the  analyst  to  insure  that  his  or  her  research  can  be 
redirected  or  rechanneled  at  critical  points,  if  necessary,  by  establishing  a  series  of 
technical  reviews,  such  as: 

•  Sub-system  reviews 

•  Functional  reviews 

•  Interim  system  reviews 

•  Major  reviews 

•  •  System  requirement  review 

••  System  design  review 

••  Software  specification  review 

••  Preliminary  design  review 

••  Critical  design  review 

••  Functional  system  audit 

••  Functional  configuration  audit 

••  Physical  configuration  audit 
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2.08.03  Can  the  prooram  be  transitioned  from  R&D  to  prototype 
development  without  complete  redesion  and/or  redevelopment? 

"Research  and  development"  is  defined  as  effort  directed  toward  gaining 
knowledge/  or  understanding/  necessary  for  determining  the  means  by  which  a 
recognized  and  specific  need  may  be  met.  In  industry,  "research"  includes 
investigations  directed  to  the  discovery  of  new  scientific  knowledge  that  have 
specific  commercial  objectives  with  respect  to  products  or  processes. 

"Development"  is  the  systematic  use  of  the  knowledge  or  understanding  gained 
from  research  towards  the  production  of  useful  materials,  devices,  systems  or 
methods,  including  design  and  prototyping. 

A  "prototype"  can  be  defined  as  a  model  suitable  for  the  evaluation  of  design, 
performance,  and  product  potential.  In  other  cases,  it  can  be  defined  as  a  version  of  a 
system  that  does  not  exhibit  all  the  properties  of  the  final  system.  It  is  usually 
somehow  lacking  in  terms  of  functional  or  performance  attributes. 

Therefore,  the  analyst  must  be  aware  of  whether  or  not  his  or  her  program  can  be 
transitioned  from  R&D  to  prototype  development  without  complete  redesign 
and/or  redevelopment  because  of  the  potential  effects  that  dollar  costs,  training  time 
lost,  implementation  delays,  etc,  will  have  on  the  program. 
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3. 


PROGRAM  PLANNING 


3.01  CoonHnatlon/Interface: 


3.01.01  Has  formal  contact  been  made  vrltb  the  user  or  user 
representative? 

The  identification  of  the  user  or  someone  who  has  been  designated  to  represent  the 
user.  But  this  must  be  done,  and  it  must  be  done  early,  or  your  efforts  will  be  for 
naught  Initially,  go  through  your  chain -of-command.  If  that  is  not  productive, 
contact  Air  Training  Command  or  the  Air  Staff  training  and  education  offices 
within  DCS  Personnel.  Areas  such  as  OJT  or  ancillary  training  are  so  widespread 
that  it  may  have  been  necessary  to  designate  someone  at  one  of  those  levels  to  act  as 
the  user  representative. 

Once  the  user  or  user  representative  is  identified,  formal  contact  should  be  made  to 
begin  what  should  be  a  continuing  dialogue  throughout  the  project  Early  and 
frequent  involvement  of  the  user  (your  "customer")  will  help  avoid  problems 
down  the  road. 


3.Q1.01.01  Has  the  user  or  user  representative  agreed  to  the  need  for 
research? 

The  project  should  probably  not  be  undertaken  if  this  question  cannot  be  answered 
affirmatively.  It  is  a  good  idea  to  get  this  agreement  in  writing  in  terms  of  a 
memorandum  of  agreement,  or  some  comparable  document 


3.01.01.02  Has  the  user  or  user  representative  "bought  into"  the  research 
approach? 

It  is  preferable  that  this  "buy-in"  be  obtained  early-on;  however,  that  is  not 
necessarily  critical  for  all  projects.  The  research  approach  and  the  user's  acceptance 
of  it  may  evolve  over  time,  particularly  for  more  complex  efforts. 


3.01.01.03  Has  a  user  representative  been  Identified  as  a  focal  point  or 
participant  on  the  research  team? 

Having  a  user  representative  on  the  research  team  would  be  the  ideal  situation,  but 


t*$so 


459 


it  may  not  be  practical  or  affordable.  It  is  essential,  however,  that  a  focal  point  be 
identified  who  can  represent  the  users'  views.  This  person  should  be  in  a  position 
of  sufficient  authority  to  make  commitments  for  the  user. 

3.01.02  Havo  Interdependencies  with  otter  protects  been  Identified  and 
Incorporated  Into  project  schedules  and  plans? 

Check  within  your  organization,  and  with  the  user  or  user  representative  to 
determine  whether  there  are  other  projects  in  progress  or  being  planned  which  will 
require  inputs  or  outputs  horn,  or  interface  or  coordination  with,  your  efforts.  This 
should  be  done  early  on  in  order  to  avoid  duplication  or  scheduling  problems  later. 
(See  question  3.05.02.) 


3.01.03  Will  ttn  technology  Interface  with  otter  types  of  tralnlng/tralnlnp 
devices? 

This  is  important  to  establish  early  on,  and  it  is  just  as  important  to  remain 
sensitive  to  it  throughout  the  life  of  the  project  HQ  ATC  is  probably  one  of  your 
better  sources  for  this  type  information,  as  well  as  your  own  organization.  There 
may  be  instances  where  interface  is  not  an  issue,  but  where  it  is  an  issue,  it  is 
essential  that  it  be  addressed. 

3.01.04  Has  coordination  been  effected  with  otter  Interested  or  affected 
agencies,  n.iu  ATC,  All,  AFIT7 

You  can't  do  much  in  the  area  of  training  without  having  an  affect  on  ATC,  or  in 
PME  without  affecting  AU,  or  in  education  without  affecting  AFTT.  The  appropriate 
organization  needs  to  be  kept  informed  throughout  the  project  And  there  may  be 
others  as  well,  such  as  the  Air  Staff,  the  AF  Occupational  Measurement  Squadron 
(AFOMS),  the  AF  Military  Personnel  Center  (AFMPC),  etc  The  important  tiling  is 
to  coordinate  your  activities  frequently  with  affected  agencies. 


3.01.05  Have  ootentlal  operational  Impacts  been  Identified? 

You  may  have  to  get  down  the  road  a  ways  before  making  this  determination,  but  it 
must  be  made  and  communicated  to  the  user  to  mitigate  adverse  impacts.  You  and 
the  user  representative  need  to  work  closely  on  this  question.  Identification  of 
operational  impacts  should  be  an  item  of  particular  interest  during  operational  test 
and  evaluation.  Impacts  may  differ  before,  during,  and  after  implementation,  and 
they  can  be  either  positive  or  negative. 
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3.01 .05.01  Will  units  possibly  face  temporary  reduced  readiness  during 
Implementation? 

This  should  be  addressed  as  a  specific  element  of  question  3.01.05  above.  If  reduced 
readiness  is  anticipated,  make  sure  the  emphasis  is  on  "temporary.” 


3.01.05.02  Will  operational  resources  (manpower,  dollars,  equipment)  be 
Impacted? 

If  your  project  will  result  in  resource  savings,  you're  probably  in  good  shape  and  you 
will  be  appreciated  by  the  user.  If  you  forecast  that  your  project  will  result  in  the 
need  for  more  resources  at  the  operational  level,  you  probably  should  reconsider  the 
whole  effort  -  it  is  doubtful  such  a  project  will  receive  user  support.  At  the  very 
least,  you  should  be  prepared  to  demonstrate  the  benefits  which  might  justify  the 
oost. 


3.01.06  Have  potential  operational  Impacts  been  discussed  with 
appropriate  operational  officials? 

This  should  be  done  as  soon  as  operational  impacts  are  identified.  The  user 
representative  can  help  you  identify  who  should  be  advised.  Don’t  yield  to  the 
temptation  to  avoid  this  one  —  to  do  so  will  hurt  not  only  your  project,  but 
follow-on  efforts  of  your  agency  ar  well. 


3.01 .07  Have  actions  been  taken  to  minimize  potential  operational 
Impacts? 

These  actions  can  take  many  forms,  and  should  be  formulated  in  dose  coordination 
with  the  user  representative.  The  use  of  comprehensive  development  and 
operational  testing  is  probably  one  of  the  better  ways  to  identify  and  minimize 
operational  impacts. 

3.02  Funding: 


3.02.01  Have  funding  source(s)  been  identified  for  this  research? 

Hopefully,  this  is  somewhat  of  an  academic  question  at  this  point.  If  a  funding 
source  hasn't  been  identified,  there  is  no  way  the  project  can  proceed.  If  the 
proposed  project  has  the  support  of  the  user,  but  no  funding  source  has  been 


identified,  steps  should  be  taken  to  find  a  funding  source.  Start  with  your 
chain-of-command  and  internal  comptroller  activities.  If  this  is  not  fruitful,  staff  a 
proposed  funding  request  to  the  user  for  funding  support  If  a  funding  source  hasn't 
been  identified  by  now,  you  can  expect  the  start  of  the  project  to  be  delayed 
considerably,  if  not  cancelled  altogether. 


3.02.02  Have  funds  been  allocated  for  this  research? 

Just  because  a  funding  source  has  been  identified  (question  3.02.01)  doesn't  mean  the 
project  has  been  funded  unless  the  funds  have  been  allocated  or  committed 
specifically  to  the  project 

3.02.03  An  allocated  funds  adequate? 

If  the  project  isn't  fully  funded,  what  can  you  do?  There  are  at  least  four  options 
The  project  can  be  cancelled,  changed  in  scope,  delayed  until  full  funding  is 
obtained,  or  fully  funded  through  other  means.  This  decision  must  be  made  in 
close  coordination  with  the  user  representative. 


3.02.04  If  funds  are  not  adequate,  have  steps  been  taken  to 
request/obtain  additional  hunting? 

In  most  instances,  this  is  probably  going  to  be  die  preferred  alternative  in  response 
to  question  3.02.03  above.  The  process  is  essentially  the  same  as  described  under 
question  3.02.01  above,  that  is,  check  your  chain-of-command  first,  then  attempt  to 
negotiate  with  the  user  for  supplemental  funding. 


3.03  Staffing: 


3.03.01  Has  an  accurate  assessment  of  manpower  requirements  to 
accomplish  In-house  research  been  accomplished? 

Hopefully,  this  was  done  as  part  of  the  "costing"  of  the  project.  It  should  be  an 
objective  determination  of  manpower  or  manhours  required  to  accomplish  the 
project,  without  regard  to  availability  of  manpower  for  the  moment  (availability 
will  be  established  later). 
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3.030)1  J1  Will  increased  manpower  be  reoulrad  to  support  the  research? 

This  is  merely  a  comparison  of  required  and  available  manpower. 


103.01.02  Can  Increases  he  satisfied  through  realignment  of  existing 
manpower  resources? 

Your  chain-of-command  must  make  this  determination  based  upon  availability  and 
priorities.  If  the  answer  is  no,  you  again  have  at  least  the  same  four  options 
addressed  above  under  Funding  (question  3.02.03).  The  project  can  be  cancelled, 
changed  in  scope,  delayed  until  manpower  becomes  available,  or  fully  funded  from 
a  manpower  standpoint  through  other  means.  (If  full  dollar  funding  is  available, 
but  adequate  in-house  manpower  resources  are  not,  consideration  can  also  be  given 
to  initiating  or  expanding  contracting  out  of  the  effort  See  question  3.03.02.) 


3.03.01 .33  Are  poteotial  user  ageodes  willing  to  reliuaulsh  manpower 
resources  on  a  temporary  basis  to  support  research  efforts? 

It  doesn't  hurt  to  ask!  If  the  project  is  of  sufficient  priority  to  the  user,  this  may  be  a 
workable  solution.  Expect  the  project,  or  at  least  some  segment  of  it  to  be  delayed, 
however.  It  normally  takes  time  to  transfer  manpower  authorizations  and  assign 
the  right  personnel  to  them. 


3.03.02  Has  a  determination  been  made  as  to  which  parts  of  the  research 
will  be  done  in-house  w ersus  contracted  out? 

This  determination  should  be  made  early  on.  If  work  is  to  be  done  by  contract,  time 
must  be  allowed  for  its  preparation,  processing  and  approval.  Criteria  to  be  used  in 
making  this  determination  include  die  availability  of  capability,  talent,  expertise, 
equipment,  and/or  facilities;  availability  of  funds;  and  cost  effectiveness. 


3.03.02.01  Is  It  more  cost  effective  to  contract  out  the  research,  or  portions 
thereof? 

This  is  a  must,  especially  if  required  capabilities  are  known  to  exist  in  both  in-house 
and  contract  sources. 
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3.03.03  Hava  in-house  resources  been  Identified  to  perform  the  research? 

This  refers  to  personnel  resources  (people,  faces,  flesh  and  blood)  as  opposed  to 
manpower  resources  (spaces,  authorizations)  referred  to  under  question  3.03.01 
above.  Have  the  people  which  the  research  requires  been  identified  by  name?  Note 
the  emphasis  on  required  —  availability  is  a  separate  issue  md  is  addressed  later. 

3.03.M  An  personnel  Identified  to  perform  the  research  adequately  and 
appropriately  skilled? 

Hopefully  this  was  taken  into  account  when  identifying  the  people  required  to  work 
on  the  project,  but  if  they're  not,  steps  need  to  be  taken  to  get  them  "up  to  speed," 
recognizing  that  this  could  delay  the  project,  or  parts  thereof,  in  addition  to  possibly 
adding  to  die  cost 


3.03.05  An  the  personnel  needed  to  conduct  the  research  available? 

If  the  answer  is  no,  you  again  have  at  least  the  same  four  options  addressed  above 
under  Funding  (question  3.02.03)  and  manpower  (question  3.03.01.02).  The  project 
can  be  cancelled,  changed  in  scope,  delayed  until  the  right  people  become  available, 
or  fully  funded  from  a  personnel  availability  standpoint  through  other  means  such 
as  borrowed  labor.  (If  full  dollar  funding  is  available,  but  adequate  in-house 
personnel  are  not,  consideration  can  also  be  given  to  initiating  or  expanding 
contracting  out  of  the  effort  See  question  3.03.02.) 


3.03.06  Is  the  organizational  placement  and  supervision  of  the  research 
appropriate? 

This  can  be  a  difficult  issue  in  a  military  organization.  Sometimes  projects  begin  in 
the  right  place,  but  as  they  mature  their  focus  changes  to  the  point  that  the  project 
and  the  people  working  on  it  become  the  illegitimate  children  in  the  organization. 
This  can  result  in  lack  of  management  attention,  reduced  priorities  or  funding, 
reduced  morale,  and  probably  a  very  unhappy  customer  (the  user).  If  this  happens, 
or  for  whatever  reason  the  project  was  incorrectly  assigned  in  the  first  place,  this 
should  be  brought  to  management’s  attention,  and  hopefully  corrected. 
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3.04 


Facilities: 


3.04.01  Are  the  facilities  to  conduct  the  research  available? 

You  need  to  think  beyond  your  own  agency  in  assessing  the  availability  of  facilities. 
Think  in  terms  of  required  user,  contractor,  and  test  facilities  as  well.  Also,  think 
beyond  office/lab  space.  For  example,  if  the  development,  procurement,  and/or  use 
of  training  devices  is  contemplated,  plans  far  their  facilitation  must  be  made  well  in 
advance. 


3.04.02  Are  necessary  supplies  and  equipment  available? 

The  same  issues  apply  here  as  for  question  3.04.01  above. 


3.05  Time-line: 


3.05.01  Has  a  time-line  been  developed? 

The  sophistication  of  the  time-line  will  depend  largely  on  the  complexity  of  the 
project,  but  even  the  simplest  of  projects  should  be  placed  on  a  time-line  or 
schedule.  The  time-line  should  be  challenging,  but  realistic  and  acceptable  to  the 
user  (see  question  3.05.03  below). 


3.05.02  Where  interdependencies  with  other  projects  have  been 
Identified,  have  time-lines  been  synchronized? 

Refer  to  question  3.01.02.  If  interdependencies  have  been  identified,  they  must  be 
taken  into  account  when  developing  your  project's  time-line.  Failure  to  do  so  can 
delay  your  or  others'  projects.  And  don't  forget  these  interdependencies  if 
unforseen  circumstances  cause  the  need  for  adjustment  to  your  time-line  as  the 
project  proceeds. 


3.05.03  Is  the  time  that  Is  necessary  for  research,  development,  and 
testing  congruent  with  users'  expectations  for  availability? 

This  should  be  established  early  in  the  planning  stages  of  the  project.  If  unforseen 
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circumstances  subsequently  cause  the  need  for  adjustment  of  the  time-line,  the  user 
representative  should  be  consulted  immediately.  You'll  have  an  unhappy  customer 
if  you  wait  too  long  and  surprise  him/her  later. 

3.05.04  Does  the  time-line  Include  the  point  at  which  you  expect  to  move 
tram  R&D  to  prototype  development? 

This  is  a  significant  and  important  milestone  to  the  user,  and  a  lot  of  advance 
planning  needs  to  be  associated  with  it  It  should  be  monitored  closely,  and  if 
schedule  changes  occur,  be  sure  that  all  affected  agencies/ personnel  are  kept  advised. 


3.06  Contracting  (If  used!: 


3.06.01  If  required,  bas  a  Request  for  Proposals  bean  written? 

The  primary  function  of  the  Request  for  Proposal  (RFP)  is  to  convey  to  industry 
exactly  what  it  is  that  the  government  (you)  wishes  to  procure.  Two  skills  are 
required  to  develop  meaningful  RFPs:  mastering  the  technological  areas  that  must 
be  covered  in  the  RFP  for  accomplishment  of  the  research;  and  understanding  the 
laws,  regulations,  and  policies  that  govern  RFP  format  and  content.  Few 
individuals  are  experts  in  both.  Consequently,  you  as  the  technological  expert  need 
:o  be  able  to  communicate  with,  and  obtain  assistance  horn,  procurement  specialists 
in  preparing  an  RFP. 


3.06.02  Has  an  appropriate  Statement  of  Work  been  developed? 

Within  a  Request  for  Proposal,  the  Statement  of  Work  (SOW)  identifies  the 
requirements  which  you  want  the  contractor  to  address  during  specific  phases  of  the 
research.  The  focus  is  on  the  contractor  and  the  SOW  language  defines  the 
minimum  required  contractor  efforts,  to  include  deliverables. 


3.06.03  Have  the  conditions  of  the  contract,  e.g.,  fixed  price,  cost  plus, 
8A  set-aside,  beeo  established? 

If  a  contract  is  to  be  used,  the  conditions  of  the  contract  should  be  established  early 
on  between  you  and  the  procurement  specialists. 
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3.06.04  Has  a  contractor  been  identified/selected  to  perform  the 
research? 

If  so,  identify  here  for  the  information  and  benefit  of  all  concerned. 


3.06.05  Are  provisions  in  place  for  contract  management? 

These  provisions  should  include  technical  as  well  as  cost  and  schedule 
management.  This  can  be  done  through  periodic  interim  progress  (or  "how-goes-it") 
reports /briefings  established  as  contract  deliverables. 


3.06.06  Is  the  contract  adequately  funded? 

V  the  answer  is  no,  you  have  at  least  four  options  to  consider.  The  project  can  be 
fully  funded  from  other  lower  priority  sources,  cancelled,  changed  in  scope,  or 
delayed  until  adequate  funds  become  available.  If  none  of  these  are  considered 
viable  alternatives,  the  distribution  of  work  between  in-house  resources  and 
contractor  should  be  reevaluated  to  determine  if  more  can  be  done  in-house. 


3.06.08  Are  procedures  la  place  for  reacting  to,  and  decision  making 
concerning,  options  raised  by  the  contractor(s)? 

Such  procedures  should  be  included  as  part  of  the  overall  contract  management 
plan,  and  the  flexibility  allowing  the  contractor  to  raise  and  address  options  should 
be  included  in  the  Statement  of  Work  in  the  Request  for  Proposal. 


4. 


TEST  AND  EVALUATION 

Testing  and  evaluation  of  a  technology  involves  empirical  data  collection  on  a 
prototype  and/or  a  final  version.  The  objective  of  this  process  is  to  assess  whether  or 
not  the  technology  works  as  intended. 

The  testing  process  should  examine  the  technology  on  two  levels.  First,  it  should 
assess  the  operation  of  the  technology  itself,  which,  in  this  section,  will  be  referred  to 
as  "system  performance".  Second,  the  testing  process  should  evaluate  the  capacity 
and  willingness  of  users  to  utilize  the  technology  for  it's  intended  purpose.  This 
concept  will  be  referred  to  as  "user  acceptability." 

Checking  the  performance  of  a  new  technology  is  usually  done  as  a  series  of  steps. 
The  first  step,  "laboratory  testing”,  is  performed  to  ensure  that  the  technology 
operates  as  intended,  and  is  reliable  over  an  extended  period  of  time.  The  second 
step  is  a  small  scale  field  trial  with  typical  users.  This  field  trial  has  three  objectives: 
to  examine  if  the  technology  can  be  used  as  designed,  if  it  is  comfortable  for  users, 
and  if  it  is  effective  (at  least  in  the  limited  and  well -controlled  environment  of  the 
field  trial).  This  field  trial  is  frequently  referred  to  as  an  evaluation  of  the  "efficacy" 
of  the  technology. 

If  the  technology  fulfills  all  requirements  for  approval  in  stages  one  and  two,  then 
the  third  and  final  step  is  a  large-scale  field  trial.  The  objective  of  this  final  field  trial 
is  to  both  examine  the  processes  and  anticipate  the  problems  that  might  occur 
during  deployment  (process  evaluation),  and  discover  whether  the  intended 
outcomes  of  die  technology  are  achieved  under  conditions  similar  to  those  that 
would  be  encountered  in  hill  deployment  (outcome  evaluation). 


4.01  System  Performance: 

For  the  following  sub-sections,  the  term  "system"  is  defined  as:  "the  set  of 
components,  developed  to  work  together  to  produce  a  training  result".  For  example, 
a  technical  system  designed  to  provide  training  for  a  maintenance  operation  may 
include:  specified  simulation  equipment,  software  for  controlling  the  simulator, 
detailed  instructions  for  using  the  simulator  and  programming  the  software,  and 
perhaps  an  information  package  for  training  the  instructors/ managers  in  using  the 
equipment.  Each  component  in  such  a  system  requires  testing  and  evaluation,  both 
individually  and  together  as  a  system. 
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4411.01  Are  there  clear  and  exact  specifications  for  how  the  technology  is 
Intended  to  operate? 

The  focus  of  this  checklist  item  is  on  what  the  technology  is  intended  to  do.  The 
criteria  and  standards  discussed  below  detail  how  these  specifications  are  to  be 
measured.  The  specifications  should  include  performance  or  process  dimensions  as 
well  as  the  intended  outcomes. 

Specifications  for  what  the  technology  is  to  do  -  and  with  what  degree  of  exactness 
and  reliability  —  should  have  been  written  prior  to  the  development  of  the 
technology  (see  Section  2.01:  Identification  of  Technologies).  These  specifications 
should  be  reviewed  and  updated  as  needed,  prior  to  any  testing  and  evaluation.  If 
technical  specifications  are  not  available,  or  have  not  been  written,  this  should  be 
done  immediately. 

Sources  of  information  for  developing  specifications  include  the  following: 

•  Technical  Developers:  They  can  indicate  the  types  of  performance 
dimensions  the  system  is  intended  to  address,  and  the  technical 
capacities  of  the  hardware  and  other  components. 

•  User  Representatives:  They  should  have  some  input  in  indicating 
what  they  want  the  technology  to  do  for  them. 

•  Other  Technical  Experts:  They  may  know  what  similar  types  of 
technology  were  able,  or  unable,  to  do  in  other  circumstances.  They 
may  also  be  able  to  offer  suggestions  on  features  that  were  desired  for 
similar  systems  but  never  implemented.  Other  experts  may  be  able  to 
offer  helpful  suggestions  about  the  new  system  based  simply  on 
knowledge  about  the  theory  underlying  its  design. 

An  important  aspect  of  developing  specifications  is  to  determine  whether  the 
technology  is  a  "complete  package  of  specified  activities”  intended  to  be  turned  over 
to  the  users  and  replicated  as  designed,  or  a  "flexible  tool"  intended  to  be  used  for  a 
variety  of  purposes  and  adapted  by  the  users  to  their  own  needs.  For  example,  a 
personal  computer  is  a  good  example  of  a  flexible  took  It  is  obvious  that  a  computer 
has  many  potential  uses,  these  include  writing  and  printing  text,  maintaining 
records,  analyzing  data,  and  running  software  for  teaching  about  technologies  or  any 
other  topic.  This  flexible  type  of  technology  may  be  more  difficult  to  evaluate, 
because  its  effectiveness  depends  on  the  appropriateness  of  the  user's  application  of 
the  system,  and  the  prior  knowledge  and  experience  that  the  user  brings  in  applying 
it.  In  order  to  be  effective,  a  tool  must  not  only  be  well -engineered  and  reliable  in 
itself  -  it  must  be  used  appropriately. 


Pitfalls  to  avoid  when  developing  specifications: 

-  Focusing  only  on  the  technical  performance  aspects  and  neglecting  the 
user’s  performance  resulting  from  the  use  of  the  technology. 

Both  under -specification  and  over-specification,  in  terms  of  the  level  of 
detail  included.  Unless  a  technology  is  very  complex  with  many 
components,  ten  to  twenty  major  performance  dimensions  should 
provide  sufficient  specifications. 


441.02  Have  criteria  been  identified  against  which  to  test  performance? 

Criteria  for  testing  performance  are  die  actual  measurements  that  will  be  taken  to 
"operationalize",  or  clearly  demonstrate  the  attainment  of,  each  specification.  In 
general,  each  technical  specification  should  have  one  or  more  criteria  for  it's 
measurement  These  criteria  may  be  technical  characteristics  of  the  hardware,  such 
as  the  operating  speed  of  a  computer  when  processing  the  training  software,  or  a 
measurement  of  user  performance,  such  as  learning  a  certain  task  with  a  specified 
degree  of  accuracy  within  a  specified  time  period  when  using  the  technology.  In 
general,  criteria  are  developed  by  answering  the  question,  "What  measurements 
will  we  take  in  order  to  know  whether  or  not  this  technology  is  performing  as 
intended?" 


4.01.03  Are  a  priori  itandards  available  for  those  criteria? 

In  general  terms,  there  are  two  major  approaches  for  making  judgments  about  the 
effectiveness  of  a  technology.  One  approach  is  to  compare  the  data  acquired  in  the 
testing/evaluation  process  against  a  priori  standards.  An  a  priori  standard  is  a 
benchmark  or  performance  goal  that  can  be  specified  in  advance,  such  as,  "Using 
this  technology,  at  least  80%  of  trainees  will  learn  the  specified  tasks  to  95%  accuracy 
within  2  hours  of  training."  The  other  approach  is  to  use  comparisons  with  data 
from  some  other  technology  or  procedure  (discussed  in  the  next  item). 

If  a  priori  standards  are  available,  or  can  be  developed,  their  use  simplifies  the 
evaluation  process,  because  measurements  from  comparison  groups  are  not 
required  for  judgments  of  effectiveness.  When  the  performance  of  the  equipment 
and/or  the  trainees  meets  or  exceeds  the  performance  standards,  then  by  definition, 
it  is  effective.  The  use  of  such  standards  is  sometimes  called  an  "accounting" 
approach  to  evaluation,  because  it  uses  logic  similar  to  the  accounting  profession.  In 
accounting,  the  acceptable  practices  and  degree  of  accuracy  necessary  for  "legally 
correct"  performance  are  set  in  advance. 
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To  establish  attainable  a  priori  standards,  there  must  be  enough  experience  with  the 
technology,  and  with  the  capabilities  of  the  users.  The  standard  should  represent  a 
goal  of  improved  performance  or  excellent  performance,  without  being  so  difficult 
to  achieve  that  developers,  trainees  or  other  users  find  it  unrealistic  If  a  standard  is 
viewed  as  unrealistic,  it  is  likely  to  be  ignored  both  when  making  decisions  and  in 
practice.  If  this  situation  occurs,  then  it  is  obvious  that  the  standards  will  not  be 
useful  for  evaluating  the  performance  of  either  the  equipment  or  the  user. 

The  use  of  apriori  performance  standards  requires  that  the  major  audiences 
involved  in  both  the  use  and  development  of  the  technology  are  in  agreement  on 
the  standards  to  be  applied,  in  advance  of  the  evaluation  data  collection.  These 
audiences  and  "stakeholders"  include:  those  who  will  make  further  decisions 
concerning  continued  development  and  deployment  of  the  technical  system,  those 
who  have  been  responsible  for  developing  the  technology,  and  representatives  of 
the  ultimate  users.  Specifying  the  standards  and  negotiating  agreement  on  their 
details  among  the  relevant  officials,  may  require  considerable  time  and  effort 

If  measurements  are  collected  on  a  new  technology  without  a  basis  for  comparisons, 
it  creates  considerable  psychological  pressure  within  the  organization  to  decide  that 
the  technology  is  "good  enough"  and  to  continue  developmental  or  deployment 
funding.  This  may  result  in  the  tendency  (with  findings  well  documented  in  the 
research  literature)  for  organizations  to  continue  sinking  development  money  into 
technologies  that  are  not  effective,  in  attempts  to  "protect"  the  investments  -  both 
financial  and  personal  -  already  made.  Decision  makers  should  be  encouraged  to  use 
test  and  evaluation  data  to  cease  investments  in  ineffective  technologies  with  no 
harm  to  their  professional  reputations  or  careen,  as  well  as  to  continue  investments 
in  effective  technologies. 

Sources  of  performance  standards: 

•  From  the  technical  specifications  of  "state  of  the  art”  technical 
equipment  similar  to  that  being  tested,  such  as  the  latest  data 
concerning  expected  capacity  and  speed  of  personal  computers. 

•  From  the  measured  past  performance  of  the  technology  (or  the 
procedures)  that  the  new  technology  is  intended  to  replace. 

•  From  professional  groups  that  specialize  in  the  relevant  content  area. 
They  may  be  able  to  specify  standards  for  minimal  acceptable 
performance  or  provide  a  definition  of  what  can  be  considered  a 
significant  increase  in  performance. 

•  From  technical  experts  in  the  content  field,  whose  experience  may 
suggest  reasonable  goals. 


Several  caveats  to  the  use  of  performance  standards  have  been  mentioned  above, 
including  the  importance  of  obtaining  agreement  on  the  specific  standards  in 
advance,  and  the  necessity  that  the  standards  be  realistic  and  achievable,  in  order  to 
be  useful.  Further,  if  performance  standards  are  used  for  larger  scale  field  testing,  it 
is  important  that  local  mediating  factors  that  are  likely  to  influence  performance  be 
considered  when  judging  the  effectiveness  of  performance  against  the  standard. 


4.01.03.01  If  answer  to  4.01.03  Is  "no",  will  the  technology  be  compared 
with  another  technology,  such  as  that  currently  used? 

The  second  approach  to  making  judgments  about  the  effectiveness  of  a  technology  is 
the  familiar  "experimental  design",  in  which  the  new  technology  or  procedure  is 
systematically  compared  with  another  technology.  The  comparison  might  be  the 
equipment  or  procedures  currently  used  in  the  Held  for  the  same  purpose,  or 
another  new  technology  that  is  also  under  development  This  sort  of  comparison 
can  sometimes  be  used  as  a  "horse  race"  to  find  the  winners  among  a  number  of 
competing  products.  The  discussion  under  items  4.01.01  and  4.01.02  of  specifications 
and  measurement  criteria  still  apply,  but  now  judgments  are  made  by  comparatively 
examining  how  well  several  technologies  reach  the  same  specifications. 

Sources  of  guidance  on  how  to  set  up  and  conduct  an  experimental  test  can  be  found 
in  the  research  methods  literature  of  each  discipline  relevant  to  the  type  of 
technology  being  tested.  The  scope  of  the  experiment  should  differ  a  great  deal 
depending  on  the  stage  of  the  performance  testing.  Initially,  small-scale 
comparisons  within  the  laboratory  setting  are  likely  to  be  needed  to  test  the  technical 
performance  of  the  innovation.  Then,  controlled  comparisons  under  field 
situations  should  be  arranged,  preferably  with  the  same  users  testing  both 
technologies.  If  different  sets  of  users  are  needed,  choosing  comparison  settings  that 
are  as  similar  as  possible.  For  technologies  that  are  intended  to  affect  an 
organization-level  operation,  such  as  new  methods  cf  recording  and  tracking  the 
training  operations  of  a  unit,  the  comparisons  may  need  to  be  a  large-scale  trial 
among  several  different  bases. 

4.01.04  Have  methodologies  for  Initial  test(s)  been  determined? 

The  preceding  sections  have  discussed  some  considerations  that  will  go  into  the 
decisions  made  about  initial  and  full-scale  testing.  This  item  is  a  summary  of  the 
range  of  issues  to  be  considered  in  designing  the  initial  test(s).  The  final 
determination  of  methodologies  will  include: 

•  An  itemization  of  the  specific  measures  that  will  be  used,  These 
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measures,  in  turn,  are  based  on  the  technical  specifications. 


•  Decisions  on  the  objectives  and  scope  of  the  initial  test  is  the  purpose 
to  demonstrate  operational  procedures  and  obtain  pilot  data?  Or  is  it  to 
obtain  a  more  thorough  test  of  effectiveness?  If  the  purpose  of  this 
stage  is  to  test  effectiveness,  then  a  choice  between  the  judgmental 
approaches  will  be  needed,  i.e.,  testing  against  a  priori  standards  or 
against  one  or  more  comparisons; 

•  Determining  how  the  data  will  be  collected  for  the  measures  to  be 
taken. 

•  Specifying  analytical  techniques  to  be  used  (if  statistical  procedures  are 
needed  to  summarize  and  synthesize  large  amounts  of  data). 


For  exploratory  or  pilot  testing,  a  number  of  approaches  may  be  applicable, 
depending  on  the  nature  of  the  technology.  These  indude  the  following: 

1)  For  hardware  components,  test  each  element  of  the  technology  individually 
to  examine  whether  it  works  as  intended.  If  the  reliability  of  the  parts  or 
consistency  of  operation  may  be  a  problem,  then  the  test  should  be  repeated  as 
many  times  as  necessary  to  check  the  reliability  of  that  type  of  equipment 
What  is  likely  to  be  the  weakest  link,  physically? 

2)  For  components  requiring  a  human  interface,  enlist  testers  who  are  not 
familiar  with  the  technology  and  provide  them  with  the  instructions  or 
training  they  would  receive  in  the  field  -  then  observe  their  attempts  to  use 
the  technology.  This  is  likely  to  require  bringing  naive  users  into  the 
laboratory  or  developmental  environment.  A  small  number  of  testers 
(perhaps  8  to  12)  is  likely  to  be  sufficient  to  check  die  operational  aspects  of  the 
technology. 

The  operational  tests  should  incorporate  such  questions  as: 

•  Are  the  instructions  sufficient  for  the  user  to  operate  the  technology? 

•  How  long  is  the  learning  time  required? 

•  What  are  the  testers'  reactions  as  they  attempt  to  use  the  innovation? 

•  Are  they  able  to  use  it  as  intended? 

•  Do  they  like  it? 


What  sources  of  difficulty,  if  any,  do  they  experience? 

How  do  they  think  the  technology  would  fit  in  their  normal  working 
environments? 

For  technologies  intended  to  deliver  training  to  improve  learners' 
performance,  initial  observations  of  all  aspects  involved  in  assessing  learning 
will  be  required.  Again  8  to  12  "typical  learners"  should  employ  the  new 
technology  and  report  their  reactions.  At  this  stage,  the  objective  is  not  yet  to 
test  effectiveness  in  improving  their  performance,  but  simply  to  document 
whether  the  learner  can  operate  the  technology,  and  to  answer  such  research 
questions  as: 

After  the  expected  level  of  instruction,  can  the  learners  function  on  the 
technology  with  the  anticipated  degree  of  independence? 

Are  the  instructions  clear? 

Can  the  learners  flow  from  one  part  of  a  learning  package  to  another  as 
needed? 

Do  they  like  using  it? 

What  types  of  difficulty  do  they  experience? 

Do  they  seem  to  make  adequate  progress  in  their  learning? 

If  the  initial  testing  will  involve  multiple  trial  users  or  learners,  it  would  be 
valuable  to  conduct  a  focus  group  (a  discussion  group  led  by  a  moderator  to 
gather  qualitative  information  about  a  given  subject)  with  each  group  of 
testers  at  the  end  of  their  testing  period.  This  group  should  be  conducted  by  an 
experienced  group  discussion  leader  to  elicit  a  thorough  recording  of  the 
testers'  experience  with  the  technology.  Some  potential  questions: 

What  did  they  like  or  not  like  about  the  innovation? 

What  problems  or  difficulties  did  they  experience? 

Does  this  type  of  technology  seem  appropriate  for  people  like  themselves? 

*  Does  it  have  gender  connotations? 

*  Does  it  have  status  connotations? 

Do  those  from  any  particular  type  of  background  or  ethnicity  feel 
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it  has  any  special  meaning  for  them? 

•  Is  this  type  of  technology  likely  to  be  accepted  into  their  work  environments? 

•  Will  the  technology  have  physical  requirements  (e.g.,  for  space,  electricity, 
temperature  control)  or  maintenance  that  are  difficult  to  meet  in  their 
environments? 

This  type  of  discussion  can  be  conducted  within  a  one  to  three  hour  period.  The 
benefit  of  a  focus  group  is  that  it  may  elicit  more  subtle  considerations  that  will 
greatly  affect  the  acceptance  and  performance  of  the  technology  in  the  field,  but  was 
not  factored  into  the  technical  design  specifications  or  procedures. 


101.05  Novo  "typical  users"  been  Identified  to  participate  In  the  Initial 
lasts? 

This  item  refers  to  the  kinds  and  sources  of  individuals  who  should  participate  in 
the  various  initial  tests  discussed  in  item  4.01.04.  These  "typical  users"  should  not 
simply  be  the  lab  staff  or  friends  of  the  developers  in  the  laboratory,  for  these 
individuals  are  likely  to  have  technical  background  and  abilities  much  different 
from  that  of  the  eventual  users.  In  selecting  the  "typical  users”  one  should  include 
all  major  categories  of  users:  trainers  or  OTJ  supervisors  who  will  be  in  charge  of  the 
technology,  learners  (if  the  technology  is  to  support  learning  directly),  training 
supervisors,  and  maintenance  and  support  personnel 

Specific  arrangements  for  test  personnel  should  be  made  with  the  appropriate 
authority  of  some  operating  units  similar  to  those  that  will  eventually  use  the 
technology.  This  may  require  TDY  to  bring  these  individuals  (about  ten  of  each 
major  type  of  user,  for  example  trainers  and  learners)  into  the  laboratory  for  them  to 
"put  the  technology  through  its  paces”.  However,  a  prototype  of  the  system  might 
be  transported  to  a  typical  user  environment  for  this  stage  of  testing. 


4J1.06  Has  Initial  testing  shown  that  the  technology  operates  as 
expected,  and  Is  eompatlbln  with  normal  user  capabilities?  (If 
"no",  return  to  development  phases.) 

This  item  refers  to  the  first  decision  point  in  the  testing  and  evaluation  process.  The 
data  collected  from  the  initial  tests  -  quantitative  results,  qualitative  comments  and 
focus  group  results  from  the  trial  users  -  should  be  carefully  assessed.  This 
assessment  should  be  performed  by  a  decision  level  or  by  an  outside  body  that  does 
not  have  a  vested  interest  in  the  continuation  of  the  technology  development 
project. 


The  information  from  initial  tests  of  the  operational  characteristics  should  be 
compiled  and  assessed.  Most  importantly,  information  about  any  problems  that 
were  revealed  should  be  analyzed  in  light  of  the  following  questions: 

•  Are  the  problems  easily  correctable  or  do  they  appear  to  involve  fundamental 
flaws  in  the  design  of  the  technology  or  in  the  "fit"  between  technology  and 
user? 

•  If  the  technology  seems  to  operate  as  intended  by  the  designers,  but  the  users 
do  not  like  it,  are  there  features  that  can  be  changed  to  make  it  more  "user 
friendly"? 

•  Is  the  technology  one  that  is  rather  foreign  to  the  intended  users'  initial  frame 

•  Is  the  decision  is  reached  to  continue  with  larger  scale  efficacy  testing,  what 
procedures  will  be  used  in  the  further  trials  to  overcome  user  opposition  in 
order  to  obtain  full  implementation  and  an  adequate  testing  of  efficacy? 


The  result  of  this  analysis  should  be  a  decision  to  follow  one  of  three  courses  of 
action: 

1.  To  stop  development  of  this  technology  ai  this  time,  because  there 
appear  to  be  either  major  problems  in  its  conceptualization  or  design, 
or  the  cost  of  making  the  necessary  design  changes  would  outweigh  the 
benefits. 

2.  To  return  to  developmental  phases  to  correct  problems,  improve 
documentation  for  users,  modify  instructional  materials,  improve  the 
training  package,  or  make  other  corrections.  Sel .  <*tion  of  this  option 
means  that  there  is  a  clearly  defined  course  of  action  for  correcting  the 
problems  identified.  This  option  may  include  making  only  minor 
modifications  before  proceeding  to  a  field  trial. 

3.  To  proceed  with  a  fuller  scale  trial  to  test  the  efficacy  of  the  technology, 
specifically  to  find  out  if  it  can  produce  the  effects  intended  for  users  or 
learners. 
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4.01  J7  Hava  appropriate  methodologies  for  ao  Initial  field  trial  (or  other 
procedure  to  test  efficacy)  been  selected? 


The  key  objective  of  this  step  in  testing  and  evaluation  is  to  assess  whether  the  new 
technology  can  produce  better  results  for  users,  tinder  carefully  controlled 
conditions.  "Better"  may  mean:  in  relation  to  a  set  of  a  priori  standards  for  the 
technical  specifications,  in  comparison  to  usual  practice  or  another  technology,  or  as 
compared  to  another  technology  (as  discussed  above  in  items  4.01.03  and  4.01.03.01). 
In  most  cases,  the  measures  of  efficacy  will  be  objective  measures  of  performance, 
such  as  the  accuracy  and  speed  of  learning  when  using  the  technology,  or  the 
accuracy  and  amount  of  time  (efficiency)  needed  to  maintain  records  using  a  new 
technical  system.  Measures  of  users'  opinions  about  the  technology  or  their 
perceptions  of  ease  of  use  are  also  worthwhile,  but  these  do  not  substitute  for 
observed  measures  of  change  in  user  performance. 

If  die  overall  design  of  the  testing  will  be  based  on  a  comparison  with  set  standards, 
the  major  concern  will  be  to  obtain  a  sufficiently  large,  representative  sample  users 
to  know  whether  the  average  level  of  performance  using  the  technology  to  know 
whether  the  average  level  of  performance  meets  the  standard.  The  initial  trials 
should  have  indicated  whether  the  level  of  variability  among  users  is  large,  to  help 
specify  the  size  of  the  performance  testing  needed.  This  testing  should  also  be  done 
in  enough  different  locations  to  include  the  different  types  of  users  coming  from  a 
variety  of  backgrounds,  to  determine  whether  all  user  sub-groups  can  meet  the 
standard  using  die  technology. 

If  the  overall  design  will  be  a  comparison  among  technologies  or  against  current 
practice,  the  specific  types  of  comparisons  must  be  defined.  The  comparison 
technology  should  be  one  that  has  the  same  purposes  as  the  test  technology,  so  that 
the  same  measurements  can  be  used  and  results  can  be  compared.  If  feasible,  it  is 
desirable  for  test  subjects  not  to  know  which  technology  or  procedure  being  tested  is 
the  "primary  candidate"  being  developed.  This  ensures  that  they  will  not  attempt  to 
influence  the  results  concerning  efficacy,  either  positively  or  negatively,  by  their 
own  feelings  about  the  technology  they  are  using. 

Another  important  factor  is  the  specification  of  the  measurement  tools  and 
procedures  that  will  be  used  to  assess  efficacy.  Usually,  these  will  have  been 
developed  along  with  the  specifications  as  part  of  item  4.01.01  and  .02,  but  they 
should  be  re-examined  at  this  point  to  be  sure  they  are  still  relevant.  The  specific 
types  of  measurements  taken  to  assess  efficacy  will  depend  on  the  nature  of  the 
technology  and  its  objectives.  If  the  design  of  appropriate  measures  is  in  question, 
then  textbooks,  reference  sources  or  experts  on  that  type  of  technology  should  be 
consulted. 


The  number  of  test  subjects  or  situations  needed  for  a  comparison  depends  on 
several  factors.  One  is  the  magnitude  of  the  differences  expected  among  the 
technologies  being  tested  —  if  large  differences  are  expected,  then  a  smaller  number 
of  subjects  are  needed  to  detect  and  confirm  the  differences  in  outcome  measures,  if 
the  differences  among  the  technologies  are  expected  to  be  small,  they  may  be  hard  to 
detect  among  the  other  sources  of  variations  in  the  measures  taken.  If  this  is  the 
case,  then  the  number  of  test  subjects  needs  to  be  much  larger.  A  second  factor  in 
determining  the  number  of  test  subjects  needed  is  the  extent  of  "naturally 
occurring"  variability  or  "noise”  among  the  test  subjects.  If  this  natural  variability  is 
minimal,  then  fewer  test  subjects  are  needed  to  detect  the  expected  differences  in 
performance  due  to  the  new  technology.  If  little  is  known  about  how  much 
variability  to  expect,  or  how  large  a  difference  might  be  produced  by  the  new 
technology,  then  pilot  data  may  be  needed  to  make  a  valid  estimate.  Statistical 
advioe  may  be  needed  to  calculate  the  sample  si2e  needed  to  detect  the  effects  of  a 
variable. 

If  a  high  degree  of  precision  is  needed  in  the  resulting  data  -  such  that  small 
differences  are  likely  to  be  important  -  then  a  randomized,  controlled  experimental 
design  ought  to  be  used.  In  this  design,  the  test  subjects  or  situations  are  specified 
and  listed,  and  sometimes  pre-tested  to  assess  their  current  levels  of  performance. 
Then  they  are  randomly  assigned  to  the  "experimental  condition"  (using  the  new 
technology)  or  to  one  or  more  "control  conditions"  (using  the  comparison 
technologies).  The  process  of  randomization  helps  to  insure  that  background 
differences  between  the  test  subjects,  which  might  affect  their  results,  do  not  differ  in 
a  systematic  way  among  the  experimental  and  comparison  conditions.  More 
importantly,  randomization  allows  the  legitimate  use  of  many  statistical  procedures 
to  make  inferences  concerning  the  effects  of  the  new  technology. 

The  proceeding  discussion  of  the  numbers -of  test  subjects  and  the  various  types  of 
experimental  designs  is  particularly  relevant  in  the  testing  of  technologies  intended 
to  influence  the  performance  of  people,  and  for  those  technologies  which  require  a 
human  operator  as  a  part  of  the  overall  technical  system.  The  performance  of 
people  is  well  known  to  vary  considerably,  even  under  normal  circumstances.  Such 
factors  as:  the  extent  of  applicable  education,  previous  experience  with  that  type  of 
task  or  equipment,  interests  and  aptitudes,  and  many  others  affect  individual 
differences  in  performance.  Therefore,  the  procedures  used  to  test  the  effectiveness 
of  new  technologies  must  be  sophisticated  enough  to  distinguish  between  human 
performance  variations  that  are  due  to  the  technology  and  those  that  are  due  to  the 
fundamental  differences  between  people. 

Furthermore,  if  the  technology  will  be  used  by  varying  groups  of  people,  the  final 
tests  of  effectiveness  should  factor  in  the  range  of  differences  between  the  test 
subjects,  and  include  sufficient  numbers  of  the  different  types  of  people,  in  order  to 
examine  whether  the  technology  will  be  equally  effective  with  a  wide  variety  of 
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users.  If  the  testing  and  evaluation  is  solely  of  hardware  technologies,  and  quality 
control  in  manufacturing  can  assure  little  variation  among  individual  examples  of 
the  same  technology,  then  experimental  testing  using  substantial  numbers  of 
subjects  or  sites  should  not  be  necessary. 

Selection  of  a  methodology  for  field  trials  of  efficacy  should  also  include  the  overall 
plans  for  analysis  of  data.  Usually,  statistical  tests  will  be  used  to  infer  whether  any 
differences  in  results,  between  the  target  technology  and  whatever  it  was  compared 
to,  are  larger  than  the  random  variation  that  would  occur  by  chance.  The 
appropriate  statistical  tests  are  related  to  the  specific  comparison  design  used, 
particularly  the  fact  of  whether  or  not  test  subjects  were  randomly  assigned. 


&01.08  Nava  locatioit(s)  for  an  Initial  flold  trial  boon  selected,  contacted, 
and  permission  obtained  for  conducting  the  test? 

This  item  refers  to  the  necessary  logistics  for  conducting  initial  field  trials.  More 
than  one  location  may  be  needed  if:  the  technology  involves  organizational  aspects 
of  the  training  system,  testing  one  technology  would  "contaminate’'  users  and  bias 
their  results  when  testing  another,  or  the  evaluation  design  requires  more 
participants  m  the  trial  than  are  available  on  one  base.  A  key  aspect  is  to  select 
location(s)  with  pools  of  potential  users  that  are  reasonably  representative  of  the 
ultimate  users,  but  still  have  some  variability  in  the  types  of  test  subjects  available. 
The  locations  to  be  directly  compared  should  be  as  similar  as  possible  in  their 
relevant  characteristics.  The  location(s)  should  also  be  dose  enough  to  the  site  of 
technology  development  so  that  the  developers  can  easily  participate  and  observe 
the  initial  trial.  Responsibilities  for  conducting  the  trial  would  best  be  served  an 
evaluation  team  that  is  separate  from  the  developers,  to  avoid  any  potential  conflict 
of  interest 

A  number  of  practical  tasks  are  involved  in  conducting  a  trial.  The  level  of 
authority  for  requesting  permission  to  conduct  the  test  will  vary  depending  on  the 
numbers  of  test  subjects  to  be  involved,  whether  the  trial  is  a  short  term  (for 
example,  can  be  completed  in  one  day)  or  longer  term  trial,  and  the  scope  of  base 
activities  affected. 

Planning  for  all  the  logistics  needed  to  conduct  the  field  trial  should  indude  at  least 
the  following  considerations: 

•  What  is  the  expected  schedule  for  conducting  the  trial,  induding 
planning  time,  preparation  of  materials,  notifications,  conducting  the 
trial  and  collecting  data,  restoring  test  sites  to  "normal",  analyzing  data, 
preparing  reports,  and  reporting. 
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•  How  will  test  subjects  /participants  in  the  trial  be  notified  of  this 
activity  and  what  will  be  expected  of  them? 

•  How  will  participants  ’  immediate  supervisors  be  notified  about  the 
trial?  Will  they  be  able  to  affect  the  schedule  for  the  trial,  so  that  it  does 
not  conflict  with  other  essential  activities? 

•  Who  will  be  responsible  for  on-site  implementation  of  the  target 
technology  and  of  any  comparison  technologies?  Does  this  person 
have  sufficient  authority  to  obtain  the  cooperation  needed? 

•  Who  will  be  responsible  for  on-site  data  collection?  Ideally,  this  person 
should  have  technical  expertise  in  the  data  collection  methods  to  be 
used  and  should  be  familiar  enough  with  the  base  to  obtain  needed 
support  Will  training  for  data  collectors  be  provided? 

•  Are  die  tools  and  instruments  needed  for  data  collection  available  in 
sufficient  numbers,  and  on  site?  Has  their  accuracy  been  calibrated? 

•  Who  is  responsible  for  monitoring  on-site  data  collection?  Is  there  a 
procedure  for  quality  control  to  ensure  adherence  to  intended 
schedules  and  procedures? 

•  How  will  the  data  be  transferred  from  the  test  locations  to  the  office 
responsible  for  analysis  and  reporting  results? 

•  Will  test  participants  be  de-briefed  about  the  trial?  Will  it  be 
immediately  or  after  the  results  are  known? 


4.01.09  Will  a  large-scale  field  trial  be  needed  to  test  for  user 
acceptability,  deployment  within  field  conditions,  and/or  reliability 
ot  performance  across  time  and  adverse  contexts? 

The  result  of  the  previous  item,  4.01.08,  should  be  a  succinct  report  with  accurate 
data  answering  whether  the  new  technology  is  able  to  produce  the  required  results, 
and  if  a  comparison  trial  has  been  used,  whether  it  performs  better  than  the 
comparison(s).  This  report  should  also  include  enough  information  about  the  trial 
for  the  audience  to  know  how  it  was  conducted.  This  means  including  good 
documentation  of  methods  and  data  to  demonstrate  that  the  technology  was 
implemented  as  intended  during  the  trial.  In  concluding  the  report,  a  decision  must 
be  made  as  to  the  future  course  of  action.  Two  judgments  are  needed: 
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•  Does  the  initial  field  trial  provide  strong  data  that  the  new  technology 
is  effective  in  producing  the  intended  results?  Is  it  sufficiently  better 
than  current  practice  to  justify  a  decision  to  move  toward  full 
deployment,  with  a  replacement  of  current  practice? 

•  Will  full  deployment  involve  changes  of  such  a  magnitude  in  the  user 
environment  that  further  data  collection  from  a  large-scale  field  trial  is 
needed  to  ensure  effective  operation  of  the  technology? 

For  most  technologies  used  to  support  field  level  training,  the  answer  to  the  second 
question  will  be  "yes",  given  that  these  technologies  nearly  always  involve 
interactions  between  the  technology  and  the  users.  This  question  should  be 
answered  "no"  only  when  the  technology  can  be  deployed  into  the  field  and  will 
work  as  intended  without  human  users  and  without  disturbing  other  ongoing 
activities  in  that  environment;  surely  an  unusual  technology!  If  the  answer  to  both 
decisions  is  "yes",  then  the  technology  is  ready  for  further  field  trials. 

The  final  testing  might  be  conducted  as  feedback  data  collected  during  the  first  year 
or  two  of  partial  deployment  Alternatively,  a  larger-scale  field  trial  of  effectiveness 
within  real  conditions  may  be  needed,  particularly  when  extensive  changes  are 
needed  in  the  field  in  order  to  fully  implement  the  technology.  Large  scale  trials 
often  use  an  organizational  entity  -  squadrons,  units,  a  particular  type  of  agency,  or 
even  whole  bases  -  as  the  basic  unit  of  analysis.  Trial  organizations  should  be 
selected  and  data  collection  and  analysis  designs  prepared  using  methods  similar  to 
those  for  the  field  test  of  performance  efficacy.  Comparison  sites  (other 
organizations)  may  also  be  needed,  for  example,  if  the  costs/benefit  ratio  of  full 
deployment  is  an  issue.  Most  of  the  issues  to  be  addressed  in  the  large  scale  field 
trial  are  discussed  in  section  4.02  on  User  Acceptability. 


4.01.10  Is  appropriate  methodological  expertise  available  for  testing  the 
efficacy  of  technical  performance  and  the  effectiveness  of  the 
technology  within  field  conditions? 

The  evaluation  of  technology  performance  is  a  multifaceted  task  requiring  the  skills 
of  staff  members  from  a  variety  of  backgrounds.  For  each  discipline,  those  who 
perform  the  testing  and  evaluation  should  be  different  staff  members  from  those 
who  do  the  technology  development,  to  avoid  any  bias  in  the  findings  or  conflict  of 
interest.  If  new  or  modified  hardware  is  involved,  then  engineering  research  and 
testing  skills  are  needed  from  the  appropriate  engineering  discipline.  If  software 
development  or  applications  are  part  of  the  technology  system,  then  appropriate 
computer  programming  or  software  engineering  skills  are  likely  to  be  required. 
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Fcr  assessing  the  human  side  of  technology  performance,  a  variety  of  skills  are 
likely  to  be  needed,  which  may  require  the  collaboration  of  several  individuals.  The 
testing  of  individual  learners'  performance,  particularly  of  cognitive  or  "thinking" 
tasks,  will  require  someone  experienced  in  educational  testing.  Specific  skills  are 
required  for  developing  test  items  to  analyze  the  learning  domains  addressed  by  the 
technology,  and  for  performing  the  statistical  analysis  to  assess  whether  positive 
outcome  effects  are  demonstrated.  Examining  the  performance  of  the  overall 
technology  system  in  a  field  situation  will  require  program  evaluation  skills.  These 
skills  are  required:  to  develop  an  overall  design  appropriate  to  the  objectives;  to 
develop  measuring  instruments  or  to  work  with  other  technical  specialists  in 
developing  measuring  tools;  to  develop  the  specifics  of  the  data  collection 
procedures  (including  training  those  who  will  collect  data);  to  manage  the  logistics 
of  a  large  scale  field  study;  and  to  employ  appropriate  statistics  when  analyzing  the 
data.  To  the  extent  that  the  new  technology  will  interact  with  the  on-going 
operations  and  activities  in  the  unit-level  environment,  the  evaluation  team 
should  include  a  person  skilled  in  analyzing  organizational  behavior  and  its  change. 
Each  of  these  people  will  bring  to  the  test  and  evaluation  team  skills  from  their  own 
training  and  expertise  to  help  understand  the  operations  and  effects  of  the  new 
technology. 


4.02  User  Acceptability 

This  section  addresses  the  evaluation  of  issues  likely  to  occur  during  full 
implementation  of  the  technology  in  the  field.  When  used  in  this  section,  the  term 
"users"  include  all  those  involved  with  the  technology  in  its  field  deployment.  This 
means  not  only  learners  and  trainers,  but  also  their  supervisors,  maintenance 
personnel  responsible  for  the  technology,  office  managers  and  agency  planners,  and 
the  command  level  officers  managing  unit-level  operations.  The  term  "users"  may 
also  include  organizational  entities,  as  a  new  technology  may  affect  the  standard 
operating  routines  for  getting  things  done  within  an  organizational  unit 


Obtaining  intensive  feedback  from  typical  users  during  the  initial  testing  and 
efficacy  trials  is  a  critical  part  of  the  process,  and  has  already  been  discussed  (in 
section  4.01).  This  initial  testing  will  provide  an  understanding  of  the  types  of 
reactions  to  be  expected  from  the  field,  and  whether  specific  activities  will  be  needed 
to  foster  user  acceptance.  If  the  technology  is  welcomed  by  users  as  a  solution  to 
their  needs,  and  one  that  is  easy  and  enjoyable  to  use,  user  acceptability  will  pose  few 
problems.  However,  if  potential  users  liked  the  "old  way"  of  doing  things,  then  user 
acceptability  may  become  a  major  issue.  For  example,  changes  that  involve  the 
preferred  roles  of  key  staff  are  likely  to  be  resisted,  such  as  a  change  for  trainers  from 
in-person,  classroom-type  instruction  to  supervising  computer-based  instruction. 
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Also,  technology  deployment  requiring  organizational-level  changes,  such  as  new 
communication  links,  supervisory  relationships  or  work  flow  patterns,  will 
definitely  require  systematic  examination  of  the  activities  needed  to  ensure  user 
acceptance  within  those  organizations. 


4.02J1  Hate  osar*'  specifications  ana  criteria  for  acceptable 
performance  been  Identified? 

As  file  discussion  of  criteria  (in  section  4.01.02)  has  indicated,  users'  concerns  may  be 
different  from  those  identified  by  the  developers  of  the  technology.  It  is  vital  to 
indude  users  of  all  die  types  indicated  above  in  creating  the  specifications  and 
criteria.  Before  a  major  field  trial,  these  items  should  be  reviewed  with 
representatives  of  eadt  type  of  user  group  to  ensure  that  all  concerns  are  addressed. 

Before  full  deployment  or  during  the  early  stages  of  full  deployment  it  will  be  vital 
to  obtain  in-depth  information  from  users  to  understand  their  reactions.  This  can 
be  done  with  focus  groups  or  open-ended  interviews  with  a  range  of  initial  users.  In 
some  cases,  it  may  be  difficult  for  users  to  articulate  their  concerns,  particularly  if 
they  involve  issues  outside  of  performance,  which  might  be  perceived  as 
"illegitimate".  Technology  that  appears  to  be  a  threat  to  one’s  statur.  or  preferred 
role  may  be  resisted,  even  if  there  is  evidence  showing  its  effectiveness.  For 
example,  many  individuals  in  higher  status  occupations  resisted  using  personal 
computers  when  they  were  first  available,  perhaps  partly  because  the  keyboard 
requires  the  use  of  typing,  previously  associated  with  lower  status  secretarial 
positions. 

Often  new  technology  may  be  advocated  in  order  to  increase  efficiency,  which 
means  that  some  positions  may  be  eliminated,  thus  threatening  jobs.  This  is  a  key 
concern  when  down-sizing  is  a  mandate  Efficiency  for  the  organization  as  a  whole 
may  mean  the  loss  of  a  valued  role,  or  even  unemployment,  to  the  specific 
individuals  performing  the  tasks  to  be  eliminated.  Sometimes,  such  concerns  are 
real  ones  -  part  of  the  cost  of  the  new  technology.  Other  times,  these  types  of 
concerns  are  unfounded,  and  efforts  should  be  taken  to  dispel  them.  If  such 
concerns  exist  and  are  not  addressed,  the  actions  of  those  who  fear  they  will  be 
affected  may  undermine  the  implementation  of  the  technology,  and  counter  the 
performance  advantages  that  would  otherwise  be  possible. 


4J2.02  Nm  decisions  bean  made  on  when  osar  acceptability  will  enter 
into  testing  and  evaluation  •  e.g*  In  tbe  lab.  In  Initial  field  tests, 
in  larger-scale  field  trials? 

Since  technology,  by  definition,  involves  the  application  of  a  knowledge-based 
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devise  by  a  user,  user  acceptability  should  enter  into  the  testing  and  evaluation  at  all 
stages,  with  increasing  emphasis  as  the  testing  and  evaluation  proceeds  from  the 
laboratory  to  the  field  environments.  Several  types  of  user  acceptance  should  be 
considered:  users'  capability  with  the  technology,  users’  concerns  regarding  its 
advantages  and  disadvantages  to  themselves,  users’  commitment  to  the  technology, 
and  user  organizations'  compatibility  with  the  technology. 

Capability  refers  to  the  technical  background  and  skills  brought  to  the 
technology  by  the  new  users.  Does  each  intended  user  have  the  ability  to  use 
the  technology?  If  motor  skills,  keyboard  typing  or  simultaneous 
manipulation  of  multiple  controls  are  needed  for  the  technology,  for 
example,  a  period  of  practice  may  be  needed  to  obtain  or  refresh  the  muscle 
skills  required. 

The  design  of  the  technology  may  also  assume  certain  background  knowledge  or 
prior  training  that  may  not  be  present  in  all  users,  or  may  have  been  forgotten. 
Even  if  the  requisite  skills  and  knowledge  are  present,  individuals  may  not  feel 
confident  that  they  will  be  able  to  use  the  technology,  and  may  be  reluctant  to 
"expose  their  ignorance." 

Concerns  refers  to  users'  beliefs,  even  fears  and  anxieties,  regarding  the  value 
of  the  technology  for  themselves  on  a  personal  level.  As  discussed  above, 
even  if  the  innovation  would  be  effective  in  the  aggregate,  users  may  fear  that 
it  will  have  negative  consequences  for  them.  Unless  the  deployment  strategy 
diminishes  these  concerns,  potential  users  may  consciously  or  unconsciously 
resist  the  new  technology. 

Commitment  refers  to  users'  motivations  and  levels  of  rewards  from  using 
the  technology.  Users'  motivations  may  come  from  both  internal  and 
external  sources.  Internally,  they  may  like  the  technology,  have  fun  using  it, 
feel  a  personal  responsibility  or  involvement  in  it,  or  find  it  an  intellectual 
challenge  similar  to  a  game  or  puzzle.  Positive  external  motivation  comes 
from  perceptions  that  implementing  the  technology  is  an  important  part  of 
one’s  job,  and  that  this  activity  is  valued  by  one’s  supervisor  and  co-workers. 
Participating  in  a  trial  of  the  new  technology  or  even  visiting  a  successful  trial 
site  can  increase  the  commitment  of  others  to  try  it  for  themselves. 

Compatibility  of  the  user  organizations  and  the  technology  refers  to  the  "fit" 
between  the  two.  Such  congruence  has  many  aspects,  including  how  the 
technology  fits  into  or  dashes  with  the  local  organizational  culture  ("how  we 
do  things  around  here"),  whether  it  is  consistent  with  the  priorities  of  local 
commanders,  whether  it  fits  into  or  requires  changes  in  unit  work  flows  and 
normal  worker  tasks,  whether  supportive  communications  are  already  in 
place,  and  so  forth.  User  acceptance  should  be  examined  at  both  the 
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individual  and  the  organizational  levels. 

Decisions  about  when  and  how  to  examine  user  acceptability  should  consider  all 
types  of  user  acceptance.  The  data  resulting  from  this  assessment,  and  any 
recommendations  far  actions  needed  to  support  deployment  will  be  heavily  affected 
by  die  types  of  user  acceptability  considered  and  measured.  It  is  a  common  mistake 
to  lump  these  dimensions  together  under  the  label  of  "user  attitudes"  or  inherent 
"resistance  to  change”.  In  fact,  many  of  the  difficulties  of  deploying  new 
technologies  can  be  traced  to  initial  incompatibilities  between  the  technology  and 
the  organizational  environment,  which  call  for  different  types  of  solutions  than 
motivational  efforts  to  improve  worker  "attitudes”. 


102L03  Have  appropriate  methodologies  been  selected  for  determining 
osar  acceptability? 

In  general,  two  quite  different  types  of  methodologies  are  available  for  examining 
user  acceptability,  both  generally  done  in  die  context  of  a  trial  or  early  stage 
deployment  of  the  technology  to  the  field.  One  is  to  use  in-person  qualitative 
evaluation  methods;  the  other  is  a  sample  survey  to  obtain  representative  data  from 
a  wide  range  of  users. 

Qualitative  (also  called  "naturalistic”)  data  gathering  procedures  emphasize 
understanding  the  situation  surrounding  the  use  of  a  new  technology  from  the 
point  of  view  of  the  user.  In  qualitative  data  gathering,  multiple  methods  are  used 
to  understand  how  the  technology  is  implemented  locally,  including 
non-participant  observation  on  site,  informal  interviews,  case  studies  of  the  events 
accompanying  the  technology  in  a  new  environment,  and  open-ended  interviews 
with  a  range  of  participants  and  stakeholders  in  several  sites.  These  methods  are 
particularly  appropriate  for  understanding  workers'  concerns  and  fears  and  for 
examining  the  degree  of  compatibility  between  technology  and  local  organizational 
operations.  Such  qualitative  examinations  are  generally  carried  out  during  the  first 
six  to  twelve  months  of  the  use  of  the  new  technology,  preferably  using  multiple 
sites  to  examine  whether  processes  and  concerns  are  generally  the  same  across  sites. 
If  large  differences  in  the  level  of  implementation  or  in  the  nature  of  users' 
concerns  occur  between  sites,  a  more  thorough  examination  may  be  necessary. 

Quantitative  survey  methodologies  for  assessing  acceptability  among  representative 
samples  of  users  are  probably  most  useful  after  the  technology  has  been  deployed  in 
multiple  sites  for  about  one  year.  This  method  can  be  used  to  check  on  the  type, 
intensity,  and  level  of  use  that  has  been  obtained  (if  other  data  are  not  available  for 
this  purpose),  to  assess  the  users  perceptions,  levels  of  commitment,  and  confidence 
about  using  the  technology,  and  to  get  their  perceptions  of  the  organization's  use  of 
the  technology.  Often,  many  informants  from  the  same  units  (or  other 
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organizational  entities)  are  included  in  the  sample  to  check  the  consistency  of 
responses  obtained  from  the  same  unit.  The  information  obtained  from  small  scale 
qualitative  interviews  may  be  used  to  construct  questionnaire  items  for  a  sample 
survey.  The  selection  of  respondents  for  the  survey  should  include  enough 
individuals  from  specific  organizational  units  to  characterize  these  units  (by 
aggregating  the  individual  data),  as  well  as  a  statistically  representative  sample  of 
the  population  of  individuals  using  the  technology. 

An  important  pitfall  at  this  point  is  to  assume  that  as  soon  as  the  technology  is 
deployed  in  the  field,  it  will  be  implemented  correctly.  There  are  many  examples  of 
technologies  shipped  to  local  sites  only  to  sit  unused,  sometimes  still  in  their 
packing  cartons,  because  issues  of  implementation  were  neglected.  Follow-up  data 
to  assess  user  acceptability  and  organizational  implementation  are  essential.  If  the 
results  reveal  lower  than  desired  use  in  the  field,  further  actions  may  be  necessary  to 
increase  the  extent  of  implementation  and,  thus,  the  effectiveness  of  the  technology. 
These  actions  might  include  further  organizational  analysis  and  change  to  enhance 
organizational  compatibility,  train-the- trainer  programs  for  increasing  user 
capabilities,  technical  assistance  for  modifying  work  roles  or  increasing 
communication,  or  obtaining  greater  support  from  local  supervisors  through 
chain-of-command  directives. 

4.02.04  Is  appropriate  methodological  expertise  available  for  evaluating 
user  acceptability? 

The  expertise  needed  for  assessing  user  capabilities  will  be  similar  to  the  skills 
needed  for  examining  the  effectiveness  of  the  technology,  with  an  emphasis  on 
program  evaluation  and  implementation  assessment  within  organizations.  These 
skills  are  most  likely  to  be  held  by  a  researcher  with  a  background  in  organizational 
sociology  or  psychology,  organizational  study  in  anthropology,  or  the  applied  aspects 
of  communications  research  or  political  science.  The  essential  element  is  previous 
experience  and  skills  in  collecting  data  within  and  about  organizations.  Data 
syntheses  will  usually  involve  combining  information  from  qualitative  and 
quantitative  analyses  to  put  together  a  picture  of  the  extent  of  technology 
implementation  in  the  field  from  the  points  of  view  of  its  multiple  users. 
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5.  IMPLEMENTATION/DEPLOYMENT  PLANNING 


5.01  Cost-EfhctlvBMJS  Analysis  Igr  Deployment 

The  analyst  must  ensure  that  the  following  tasks  are  accomplished  prior  to  making 

a  new  system  deployment  recommendation: 

•  A  comprehensive  evaluation  of  projected  benefits  (i.e.,  personnel  reductions) 
is  completed  that  justifies  the  system; 

•  Potential  work  loads  are  analyzed  to  determine  needed  capacity,  processing, 
and  performance  requirements  for  the  system; 

•  Hardware  alternatives  are  evaluated  and  the  best  and  most  cost  effective  one 
is  selected;  and 

•  An  operational  test  and  evaluation  of  the  complete  system  is  successfully 
completed  to  determine  its  operational  effectiveness  and  suitability. 


5.01.01  Has  tho  production  cost  per  onlt  of  the  technology  bean 
determined  for  potential  large  scale  deployment? 

An  important  first  step  in  choosing  the  best  system  design  alternative  is  identifying 
the  work  load  requirements  that  the  system  must  satisfy.  An  analysis  of  projected 
work  loads  helps  define  the  capacity,  processing,  and  performance  needs  of  a  system 
prior  to  deployment  A  cost/benefit  analysis  of  each  alternative  that  meets  these 
requirements  should  then  be  performed  to  determine  the  potential  production  cost 
per  unit.  These  steps  help  ensure  that  the  best  alternative  is  chosen. 


5.01.02  Has  flia  average  effectiveness  of  tho  technology  boon  determined 
under  "real  world"  conditions? 

Military  regulations  for  system  development,  test,  and  evaluation  reinforce  the 
importance  of  testing  a  completed  system.  These  regulations  state  that  the  approval 
of  a  system  for  deployment  should  be  supported  by  fully  developed  and  tested 
computer  programs.  These  regulations  also  state  that  deployment  should  only  take 
place  after  the  successful  completion  of  both  a  development  test  and  evaluation 
stage  and  an  operational  test  and  evaluation  stage. 

The  purpose  of  the  system  development  and  testing  phase  is  to  discover  and  correct 
problems,  and  to  determine  if  the  entire  system  works  as  intended.  To  field  a 
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system  when  it  is  only  partially  developed  and  tested  increases  the  risk  of  problems 
occurring  later  in  the  development  cyde  when  they  are  more  costly  and  more 
difficult  to  correct 


5.01.03  Have  the  resources  needed  tor  effective  large  scale 
Inplenentatloo  been  Identified? 

The  final  result  of  the  developmental  process  is  the  funding  of  the  implementation 
of  the  project  within  the  resource  constraints  of  the  organization.  Use  of  these 
resources  must  be  planned  and  coordinated  early  in  die  life  cycle  process.  Types  of 
resources  required  for  an  effective  large  scale  implementation  axe  recurring  costs  for 
operation  and  maintenance,  such  as  : 

•  Training 

••  Equipment 
••  Services 
••  Facilities 
••  Personnel 

•  Industrial  Facilities 

••  Construction/Conversion/Expansion 
••  Equipment  Acquisition  or  Modernization 
••  Maintenance  and  Repair 

••  Materials,  Supplies,  Utilities,  and  Other  Services 

•  Personnel 

••  Crew  Pay  and  Allowances 
••  Maintenance  Pay  and  Allowances 
••  Indirect  Pay  and  Allowances 
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5.01.04  Does  the  Increase  In  average  effectiveness  Justify  the  cost  of 
production  and  deployment  of  the  technology? 

To  reduce  the  risk  of  fielding  systems  that  do  not  work  as  intended  or  cost  more 
than  necessary,  the  military  requires  full  development  and  testing  and  a  complete 
analysis  of  alternatives  before  a  system  is  deployed.  Lack  of  knowledge  of  the 
increase/decrease  in  effectiveness  does  not  justify  taking  shortcuts  and  unnecessary 
risks  in  deploying  the  system. 


5.02  Implementation  Management 

Implementation  management  is  very  important  to  the  analyst  because  after  all  the 
science,  invention,  development,  selling  and  adopting,  die  new  technology  has  to  be 
implemented,  or  all  the  effort  is  wasted. 


5.02.01  Has  a  transition  plan  between  the  developer  and  user  been 
developed  and  approved? 

A  transition  plan  must  be  developed  between  the  developer  and  the  user  that 
includes  consideration  of  the  following  issues  and  events: 

•  A  system  test:  Performed  to  determine  the  acceptability  of  the  system’s 
functioning  and  data  to  the  system  users. 

•  Formal  user  acceptance:  The  analyst  must  obtain  a  written  sign-off  by 
the  designated  user  acceptor.  This  sign-off  indicates  that  the  functions 
and  data  provided  by  the  system  meet  the  user's  requirements. 

•  Generate  production  initiation  norice:  All  organizations  affected  by  the 
system  implementation  date  must  be  notified.  This  notification  must 
state  how  the  new  system  will  affect  them,  and  what  must  be  done 

for  preparation.  This  notice  will  include: 

••  Schedule:  Give  the  date  and  time  for  system  implementation 
and  phased  activities,  including  any  parallel  operations  that  are 
planned. 

••  Effect:  Give  a  summary  of  the  effects  of  the  new  system,  and 
explain  the  differences  between  the  old  and  new  systems. 

••  Coordination:  Specify  what  activities  must  be  completed  by  the 
user  to  assist  in  system  implementation,  including  training 
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system  users. 

••  Contacts:  list  personnel  contacts  for  the  system. 

Prepare  software  change  control  procedures:  These  procedures  are 
used  to  maintain  software  integrity,  minimize  life  cycle  software  costs, 
prevent  unnecessary  or  marginal  changes,  establish  change  priorities, 
assure  prompt  action  on  changes,  document  these  changes,  and  control 
the  release  of  changed  software  and  documentation. 


Obtain  production  acceptances  The  system  is  considered  operational 
and  no  longer  developmental  after  production  acceptance  by  the 
system's  custodians. 


Prepare  operation  instructions:  These  instructions  are  for  running  the 
ap^ication  systemu  Personnel  should  already  be  trained  on  the 
operation  of  the  equipment,  operating  system,  peripheral  devices,  and 
non-application  software. 


5.02.01.01  Has  an  aoencv/ofllca  baaa  designated  to  manage  the 
Implementation  process? 

The  analyst  must  determine  what  agency /office  has  been  designated  to  manage  the 
implementation  process.  This  office  is  responsible  for  certifying  that  the  new  system 
performs  according  to  the  user's  functional  and  data  requirements. 

5.02.01.02  Has  analysis  baaa  conducted  to  determine  what  operational 
procedures  or  elder  activities  will  be  changed  or  replaced  by  the 
new  technology? 

This  is  largely  a  user  responsibility;  however,  the  analyst  can  expect  to  be  asked  for 
input  In  addition  to  what  procedures  and  activities  are  affected,  the  analyst  should 
address  where,  when,  and  how  the  new  activities  will  occur.  If  current  procedures 
or  activities  are  documented  in  regulations,  pamphlets,  operating  instructions,  etc., 
these  documents  will  require  changes  or  replacement  (see  5.03.07). 


5.02.01.03  Will  uss  ol  the  technology  free  operational  equipment  that  was 
previously  reuulrod  to  conduct  certain  phases  ol  training? 

One  measure  of  implementation  effectiveness  is  the  measurement  of  any 
improvements  in  equipment  utilization.  The  analyst  must  realize  that  one  of  the 
issues  involved  in  effective  implementation  is  whether  or  not  the  use  of  the  new 
technology  free  operational  equipment  that  was  previously  used  to  conduct  certain 
phases  of  training. 


5.02.01.04  Have  the  activities  needed  to  Implement  tti8  new  technology  been 
Identified? 

The  analyst  trust  develop  an  implementation  plan  for  the  new  technology  that 
oould  be  designed  as  follows: 

•  Implementation  strategy.  Using  the  acquisition  strategy,  the  current 
system  (if  one  exists),  and  the  project  plan,  the  analyst  must  decide: 

••  if  the  implementation  will  be  gradual  or  "phased”; 

••  if  parallel  operations  are  necessary; 

••  if  a  conversion  is  necessary;  and 

••  what  is  the  best  implementation  method. 

•  Timing.  The  analyst  must  determine  when  to  implement  the  system 
based  on  information  such  as  the  new  fiscal  year,  policy  effective  date, 
resource  availability,  etc.  An  implementation  schedule  must  also  be 
developed. 

•  Responsibility.  The  analyst  must  identify  responsibilities  for  activities 
such  as  site  preparation  and  acceptance,  initial  procurement  of 
supplies,  and  implementation  coordination. 

•  Affected  organizations.  The  analyst  must  identify  organizations 
affected  by  the  implementation  and  notify  them  of  their  preparation 
responsibilities. 

•  Operations.  He  must  define  operational  functions,  and  make  resource 
estimates  and  commitments. 
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5.02.01.05  Has  It  been  determined  how  Items  will  be  supported?  (e.g., 
supportabHlty  funds,  in-house  or  contract) 

It  is  very  important  to  the  analyst  that  the  determination  be  made  early  in  the 
project  life  cycle  as  to  how  the  new  technology  will  be  supported  once  it  reaches  the 
implementation  phase.  This  allows  for  proper  resource  planning,  budgeting,  and 
timeliness  in  the  implementation  phase. 


5.02.02  Will  potential  users/managen  of  the  technology  need  to  be  trained 
In  Its  use? 

It  must  be  decided  if  potential  users /managers  of  the  technology  need  to  be  trained. 
If  there  is  a  training  requirement,  the  analyst  must  develop  a  user  training  plan, 
such  as: 

•  Training  plan  scope  and  content  The  analyst  must  identify  the 

equipment,  software,  and  procedural  training  needed  for  management, 
administration,  development,  user,  and  operation  personnel. 

•  Personnel  training  requirements.  The  analyst  must  identify 

training  needs  and  when  the  training  should  be  conducted. 

•  Presentation  methods.  It  must  specified  if  the  training  will  be  formal 
(classroom)  or  on-the-job,  and  if  all  similar  training  can  be  conducted  at 
one  time  or  if  it  must  be  phased. 

•  Training  space  and  equipment.  It  must  be  determined  if  the  space 
required  for  training,  any  special  training  equipment  (such  as 
audio-visual),  and  any  technical  support  equipment  is  available. 

•  Funding.  Any  training  costs  must  be  identified  and  a  budget  prepared. 
This  budget  should  indude  costs  for  instruction,  materials,  travel,  etc 

•  Training  team.  The  person(s)  responsible  for  negotiating  and 
administering  training  must  be  identified. 

5.02.03  Will  users  and  supervisors  have  time  available  to  learn  to  use  the 
new  technology? 

The  analyst  must  determine  whether  or  not  users  and  supervisors  have  the  time 
available  to  learn  to  use  the  new  technology.  If  training  is  required  and  time  is 
available,  the  analyst  should  establish  a  training  plan  as  outline  in  paragraph  5.02.03 
above. 
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Implementation  Resources: 


5.03.01  Have  funding  sources  been  designated  for  ail  aspects  of 
deployment  and  Implementation,  including  hardware  and  software 
purcbase  (as  needed);  training  of  users;  local  supplies, 
equipment  and  facilities;  and  maintenance  on-site? 

Although  these  funds  and  their  sources  are  probably  largely  within  the  purview  of 
the  user,  the  researcher  should  take  an  interest  in  verifying  their  availability.  If 
implementation  resources  aren't  fully  funded,  the  project  might  have  to  be 
cancelled,  changed  in  scope,  or  delayed  until  full  funding  is  obtained. 

A  good  place  to  start  in  verifying  the  availability  of  funds  is... 


5.03.02  Is  then  a  manpower  Impact  (plus  or  minus)  associated  with 
Implementation  of  the  technology? 

This  should  be  determined  before  implementation,  probably  as  a  result  of  test  and 
evaluation.  Once  known,  it  should  be  communicated  to  the  activity(ies)  impacted 
in  order  that  adjustments  can  be  made  in  a  timely  manner.  If  there  is  an  additive 
manpower  cost,  sources  will  have  to  be  found  for  those  additional  authorizations, 
and  that  can  take  time  and  extensive  effort  If  there  are  manpower  savings,  you'll  be 
someone’s  hero! 


5.03.02.01  Is  the  manpower  Impact  in  user  agencies? 

This  and  foe  following  question  should  normally  be  addressed  together,  for  an 
increase  in  one  activity's  manpower  may  be  offset  by  a  corresponding  decrease  in  the 
other.  This  requires  a  coordinated  effort  between  you,  foe  user  representative,  and 
the  affected  training  activity. 


5.03.02.02  Is  the  manpower  Impact  in  ATC  or  other  training  agency? 

See  question  5.03.02.01  above. 
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5.03.03  Is  appropriate  staffing  available  Yor  managing  implementation, 
local  use,  maintenance,  and  updating  the  technology? 

This  is  basically  a  user  issue,  but  you  should  assist  by  identifying  the  management 
requirements. 


5.03.04  Are  needed  facilities,  equipment,  and  supplies  available  and 
assigned  at  Yhe  unit  level? 

This  is  basically  a  user  issue,  but  you  should  assist  by  identifying  the  facility, 
equipment,  and  supply  requirements. 


5.03.05  Has  an  appropriate  time-line  been  designed  for  earning  out 
Implementation  processes? 

The  sophistication  of  the  time-line  will  depend  largely  on  the  complexity  of  the 
project  and  its  implementation,  but  even  the  simplest  of  projects  should  be  placed 
on  a  time-line  or  schedule.  The  time-line  should  be  developed  in  close  coordination 
with  the  user  representative. 


5.03.06  Is  tho  production  avallabllftf  of  fha  technology  and  a!!  supporting 
equipment  congruent  with  fha  tfma-fino  established? 

If  it  is  not,  actions  need  to  be  taken  to  synchronize  schedules,  either  through 
expedited  production  (which  may  be  costly),  or  adjustment  of  the  time-line.  This 
issue  must  be  worked  in  dose  coordination  with  the  user  representative. 


5.03.07  Is  the  use  of  the  new  technology  compatible  with  existing 
regulations  and  procedures? 

If  compatibility  does  not  exist,  new  regulations  and/or  procedures  may  have  to  be 
written,  staffed,  and  produced.  At  the  very  least,  it  can  be  expected  that  changes  to 
existing  regulations/procedures  will  probably  be  required.  In  any  case,  the  user  will 
look  to  you  as  the  expert  who  can  assist  in  the  drafting  of  any  new  or  changed 
procedures. 
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6.0  DOCUMENTATION  AND  FOLLOW-UP 

6.01  Documentation  of  Dot elopmont  Process 

6J1.01  Have  the  decision  and  development  processes  for  the  technology 
been  documented  to  specify  what  lessons  were  learned  during 
development  7 

The  primary  focus  of  a  research  and  development  effort  is  to  produce  some  sort  of 
gain  in  efficiency  or  productivity.  As  the  investigation  moves  forward,  decisions  are 
made  such  as:  which  avenues  of  research  should  be  pursued,  what  shape  the  final 
product  should  take,  and  when  and  how  the  final  product  should  be  implemented 
and  so  forth. 

Recording  these  decision  and  development  processes,  as  trivial  as  they  may  seem,  is 
productive  for  two  reasons.  First,  it  offers  a  guide  to  those  who  wish  to  work  with 
the  technology  in  the  future.  The  variables  involved  in  any  system  are  always 
changing,  and  any  perceived  "maximum”  will  not  be  as  permanent  as  it  may  seem. 
In  the  future,  researchers  may  want  to  update,  expand  on,  or  reinvent  the 
technology.  For  this  purpose,  a  record  of  the  rationale  behind  all  decisions  in  the 
original  research  effort  should  be  maintained.  Furthermore,  a  second  reason  that 
such  a  record  may  prove  useful  is  in  the  unfortunate  event  that  a  new  technology 
fails  to  accomplish  its  purpose.  If  this  occurs  a  record  of  decisions  will  be  critical  to 
understanding  why  the  technology  failed.  This  record  will  also  expedite  the  effort  to 
find  a  solution  to  the  problem. 

6.01.02  Have  lines  of  research  that  were  not  productive  been  documented 
to  prevent  needless  replication  ? 

Documenting  those  lines  of  research  that  were  followed  in  the  original  effort  but 
found  not  to  be  productive  should  be  documented  as  welL  This  will  allow  future 
researchers  to  better  understand  the  process  of  thought  that  directed  the  effort  in  the 
current  project,  and  will  avoid  any  duplication  of  effort  if  any  attempts  are  made  at 
improving  the  technology  in  the  future. 

6.02  Documentation  for  Technology 

6.02.01  Has  documentation  been  written  for  the  new  technology  ? 

More  often  than  not,  the  creation  of  training  manuals,  maintenance  guides,  and 
other  documentation  for  new  technologies  are  given  less  attention  than  other  areas 
in  the  research  and  development  effort  Although  this  aspect  of  implementing  a 
new  technology  may  not  be  quite  as  exciting  as  the  more  investigative,  hands-on 
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development  effort,  it  is  critical  to  the  success  of  the  new  technology.  The  efficient 
use  of  any  technology  is  entirely  dependent  on  the  users  understanding  of  the 
process  involved  in  using  the  technology  to  accomplish  the  activity. 

Many  of  us  have  seen  that  simply  giving  a  computer  to  a  worker  does  not 
immediately  increase  that  workers  productivity.  There  is  a  time  lag  due  to  the 
resulting  "learning  curve"  -  the  time  required  for  the  worker  to  learn  how  to  use 
the  computer  to  accomplish  the  activity.  This  is  true  for  all  activities  that  are  new  to 
the  person  working  with  the  technology.  Good  user  documentation  is  the  key  to 
accelerating  the  learning  curve.  For  this  reason,  documentation  should  be  given 
careful  consideration. 


6.02.02  Is  documentation  available  for  different  types  or  levels  of  users? 

A  useful  strategy  for  accelerating  the  learning  curve  mentioned  6.02.02  is  to  provide 
documentation  for  users  at  different  levels.  This  allows  advanced  users  to  operate 
at  the  level  at  which  they  are  comfortable.  Further,  one  can  also  provide  new  users 
with  information  about  the  technology  in  a  format  that  is  not  intimidating.  Often, 
new  technologies  will  be  intimidating  to  new  users.  Unfortunately,  those  people 
that  work  closely  with  the  technology  in  the  development  stages  often  find  it 
difficult  to  adopt  the  perspective  of  the  new  user  for  purposes  of  creating  a  simple 
clear,  practical  guide  to  the  new  technology.  Consulting  a  new  user  is  often  useful 
in  accomplishing  the  creation  of  an  effective  introductory  level  manual. 


6.02.03  Has  documentation  been  tested  for  clarity  and  usability  amono 
the  intended  users? 

Regardless  of  the  level  at  which  the  content  is  presented,  such  as  beginner  or 
advanced,  the  information  should  be  presented  in  a  clear,  understandable  fashion. 
As  stated  in  item  6.02.C7  <*bove,  those  people  that  work  closely  with  the  technology 
in  the  development  stages  often  find  it  difficult  to  separate  themselves  from  their 
advanced  understanding  of  the  system  in  order  to  ensure  that  they  are  not  assuming 
any  prior  knowledge  on  the  part  of  the  reader.  Consulting  a  user  representative,  if 
one  is  available,  is  a  valuable  way  of  verifying  the  clarity  of  the  documentation. 


6.02.04  Is  documentation  available  for  maintaining  the  technology? 

All  systems,  tools  or  technologies  that  may  require  maintenance  should  have 
documentation  providing  the  expected  frequency  of  maintenance  activities,  what 
activities  should  be  performed,  how  they  should  be  performed,  and  who  should 
perform  them. 


Although  maintenance  activities  for  some  systems  are  performed  by  specialized 
personnel,  in  most  situations,  it  is  advisable  to  provide  maintenance 
documentation  as  reference  to  those  that  will  be  working  closely  to  with  the 
technology.  This  allows  for  both  the  ability  to  do  simple  troubleshooting  and  avoid 
extended  down  time,  and  also  allows  advanced  users  to  increase  their 
understanding  of  their  tools. 


6.03  Follow-up 

6.03.01  Are  mere  provisions  for  follow-up  data  collection  to  determine 
whether  the  technology  Is  operating  In  the  field? 

Verification  of  system  performance  is  a  critical  part  of  the  technology  development 
process.  Appraisals  of  field  performance  ensure  efficiency  by  catching  weaknesses  in 
new  equipment  before  they  impede  progress  in  other  areas.  Making  provisions  for 
follow-up  data  collection,  and  informing  users  and  their  supervisors  that  such  an 
effort  is  to  be  arranged  will  allow  this  data  to  be  used  to  optimize  for  maximum 
performance. 

6.03.02  is  the  technology  being  used  as  Intended? 

Part  of  the  follow-up  data  collection  process  described  in  item  6.03.01  should  involve 
a  verification  that  the  equipment  is  being  used  as  intended.  It  is  important  to 
understand  that  improper  use  of  equipment  can  take  pla~e  in  very  subtle  ways.  The 
type  of  misuse  that  is  most  common  is  not,  for  example,  a  high-speed  modem  being 
used  as  a  door-stop,  or  a  cellular  phone  being  used  as  a  paperweight  The  most 
common  type  of  misuse  comes  from  the  failure  of  the  user  or  the  developer  to 
properly  communicate  the  way  in  which  the  technology  should  be  used.  This  is  the 
type  of  situation  where,  for  example,  a  user  tirelessly  repeats  a  long  sequence  of 
keyboard  actions  to  make  the  system  perform  a  certain  function,  unaware  that  the 
machine  has  a  feature  which  allows  for  long,  repeated  key  sequences  to  be  stored  in 
memory  and  executed  with  a  single  touch. 

It  is  also  important  to  be  aware  that  on  some  occasions  users  find  ways  to  make  the 
technology  perform  even  better  than  was  intended  in  the  design  process.  If  this  is 
indeed  the  case,  these  improvements  should  be  communicated  to  the  original 
development  team  and,  if  found  to  be  safe  and  effective,  spread  to  other  users.  This 
type  of  user  innovation,  when  performed  in  an  intelligent,  safe  and  non-hazardous 
manner,  should  be  encouraged  and  rewarded. 
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6.03.03  is  there  evidence  of  user  satisfaction  with  the  products? 

Verifying  user  satisfaction  can  be  accomplished  through  surveys,  on-site  inspection, 
and  informal  interviews  with  users  and/or  supervisors.  User  satisfaction  ensures 
that  the  implementation  of  the  technology  will  be  smooth  and  successful,  and  that 
expected  increases  in  efficiency  and  productivity  will  be  gained.  Verification  of  user 
satisfaction  also  ensures  that  the  developers  of  the  technology  will  be  perceived  as 
quality  conscious.  This  will  increase  the  likelihood  that  the  user  will  maintain 
contact  with  the  developers  in  the  future  about  this  and  other  systems,  ensuring  a 
good  communication  between  the  two  interdependent  parties. 

6.03.04  Are  changes,  revisions,  or  modifications  required? 

If  the  follow-up  investigation  shows  that  changes,  modifications  or  revisions  are 
required,  then  a  report  should  be  submitted  back  to  the  original  development  team 
(or  a  comparable  group  if  that  is  not  possible)  for  analysis.  It  is  important  that  the 
redesign  take  place  as  soon  as  possible  in  order  to  maintain  the  implementation 
process.  If  too  much  time  is  spent  on  analyzing  the  need  for  revisions,  then  the 
enthusiasm  and  optimism  created  by  the  original  implementation  will  be  lost.  It  is 
also  important  to  document  the  revisions  made  and  append  this  information  to  the 
original  report  on  the  research  and  development  process. 

6.03.05  Have  procedures  bean  established  for  production  of  meaningful 
progress  reports? 

Progress  reports  on  the  first  significant  period  of  use  for  a  new  technology  are 
beneficial  for  both  users  and  developers.  Most  new  systems  will  prove  or  disprove 
their  usefulness  and  sustainability  within  6-12  months.  For  this  period  (or  more  or 
less  as  required)  progress  reports  will  allow  the  developers  to  monitor  the  use  of  the 
technology  in  this  critical  period.  A  formal  procedure  should  be  established  for 
these  progress  reports  to  be  submitted:  the  appropriate  user  and/or  maintenance 
representative  should  be  appointed,  a  time  period  for  the  reports  (monthly, 
quarterly,  etc.)  should  be  decided  upon,  and  a  listing  of  the  critical  parameters  to  be 
measured  should  be  provided  to  those  responsible.  Developers  should  also  allow 
time  for  feedback,  even  if  minimal,  to  be  provided  to  those  who  write  these  reports, 
to  help  maintain  their  interest  and  motivation  in  monitoring  the  new  technology. 


